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SECTION  1.  SUMMARY 


1.1  BACKGROUND 

Test  design  and  planning  for  modem  Command,  Control,  Ccrnrnunications  and 
Intelligence  (C  i)  systems  is  becaning  an  increasingly  complex  task.  More 
sophisticated  systems  are  requiring  more  ccripl ex  testing,  in  an  environment 
with  tighter  budget  constraints.  Modem  technology  imposes  new  demands  on  the 
tester  indirectly  through  more  ccrrplex  security,  safety,  and  environmental 
considerations.  Ihe  result  is  that  testing  is  rapidly  reaching  a  point  where 
the  expertise  required  is  too  great  for  any  one  individual  to  handle 
effectively.  By  the  time  expertise  is  acquired  in  any  one  area,  the 
individual  may  retire,  leave,  or  transfer  out  of  the  organization. 

The  U.S.  Army  Electronic  Proving  Ground  (USAEPG)  has  positioned  itself  to 
alleviate  sane  of  the  problems  faced  by  today's  test  officer,  by  exploiting 
seme  of  the  very  technology  which  is  partly  responsible  for  this  dilemma: 
artificial  intelligence  (AI)  and  the  much  improved  microcomputer.  Previous 
investigations  at  USAEPG,  sponsored  by  the  Department  of  Defense  (DoD) 

Software  Technology  for  Adaptable,  Reliable  Systems  (STARS)  program  (reference 
1) ,  identified  some  aspects  of  AI  which  were  sufficiently  mature  to  insert  in 
test  tools.  One  of  these  technologies,  AI  expert  (or  knowledge-based) 
systems,  was  explored  in  depth. 

Curing  the  earlier  projects,  including  phase  I  of  this  investigation, 
prototype  expert  systems  were  developed  to  demonstrate  capabilities  and 
potential  benefits.  One  of  the  first  systems  built  to  assess  the  suitability 
of  AI  technology  for  a  proposed  application  is  still  being  used  to  screen  new 
proposals  to  eliminate  those  problems  which  are  best  addressed  with 
conventional  analysis  methods.  After  the  in-house  skills  were  developed  to 
build  expert  systems  and  differentiate  between  good  and  poor  applications,  a 
number  of  workshops  were  conducted. 

The  workshops  produced  many  good  ideas  for  expert  system  applications. 
Most  applications  were  implemented  during  the  workshops  as  "demonstration" 
level  systems.  A  smaller  number  have  evolved  into  more  robust  "prototype" 
versions.  However,  all  of  the  systems  shared  the  characteristics  of  being 
both  developed  on,  and  used  in,  a  microcomputer  environment.  (Some  of  the 
more  sophisticated  applications  built  on  other  funded  projects  at  the  USAEPG 
have  been  more  suitable  to  implementation  on  specialized  AI  machines.)  The 
viability  and  cost  effectiveness  of  these  microcomputer-based  expert  systems 
was  shown  during  phase  I  of  the  investigation  (reference  2) .  USAEPG  continued 
to  exploit  this  successful  AI  application  methodology  during  phase  II,  whose 
efforts  are  documented  below. 

1.2  OBJECTIVE 

The  objective  of  this  investigation  was  to  provide  the  test  officer  with 
automated  support  tools  by  inserting  AI  technology  in  appropriate 
applications.  Objectives  for  the  development  of  these  tools  included: 

a.  Orientation  teward  the  test  officer  as  pr inary  user. 
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b.  Wide  usability  to  satisfy  the  needs  of  the  approximately  100  test 
officers  at  the  USAEPG. 

c.  Ready  availability  (microcomputer  based) . 

d.  Reduction  in  time  to  perform  a  given  task  and/or  improved  quality  of 
the  result. 

e.  Education  of  the  user  (test  officer)  in  addition  to  merely  providing 
a  solution. 

Finally,  as  testers,  another  objective  was  to  continue  to  identify  test 
methodologies  for  the  test  and  assessment  of  systems  containing  AI. 

1.3  SUMMARY  OF  PROCEDURES 

Lessons  learned  from  earlier  work  on  expert  system  development  were 
applied  to  restructure  the  original  proposed  approach.  Rather  than  develop  a 
single  test  officer  tool  on  the  one  available  AI  machine,  an  approach  more  in 
consonance  with  the  objectives  was  established.  This  approach  called  for  the 
development  of  a  number  of  small  tools,  rather  than  risk  all  of  the  available 
resources  on  the  success  or  failure  of  a  single  large  tool.  Hie  development 
of  smaller  tools  hosted  on  microcomputers  also  provided  a  more  flexible  means 
of  adjusting  to  resource  constraints,  while  still  benefiting  from  the 
technology. 

This  modified  approach  provided  prototype  versions  of  the  Test  Plan 
Drafter  (TPD)  and  Environmental  Impact  Assessment  (EVA)  systems.  From  this 
initial  base,  new  ideas  were  developed  in  the  areas  of  meteorological  support, 
budget,  security,  contract  monitoring,  and  supporting  tools.  Systems 
addressing  these  problem  domains  were  developed  using  the  workshop 
methodology:  problem  domain  experts  and  knowledge  engineers  were  paired  to 
develop  AI -based  test  officer  support  tools. 

Finally,  the  issue  of  testing  AI  systems  was  investigated  further;  first, 
because  the  development  of  expert  system-based  tools  require  an  in-house  test 
philosophy;  and  second,  because  test  items  employing  AI  technology  are  likely 
to  appear  in  the  near  future. 

1.4  SUMMARY  OF  RESULTS 

A  number  of  AI  expert  systems  were  developed  to  aid  the  test  officer  in 
performing  duties  associated  with  testing.  With  respect  to  the  application 
objectives  outlined  above,  these  systems  satisfied  those  objectives  as 
follows. 


a.  The  knowledge  domains  of  the  expert  systems  centered  on  areas  of 
expertise  of  which  an  experienced  test  officer  would  be  cognizant,  but  not 
necessarily  an  expert.  In  other  words,  a  test  officer  might  be  familiar  with 
certain  security  or  contract  monitoring  requirements,  but  would  still  require 
considerable  consultation  with  a  domain  expert  to  satisfy  the  requirements  for 
a  new  test.  The  systems  built  during  this  phase  of  the  investigation  were 
intended  to  assist  test  officers  by  providing  the  preliminary  advice  normally 
obtained  from  the  domain  expert  during  test  planning. 
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b.  Most  of  the  systems  developed  are  still  in  the  evaluation  phase  and 
therefore  have  been  installed  on  a  limited  number  of  computer  systems.  A 
future  consid  ration  when  these  systems  emerge  from  the  prototype  stage  will 
be  to  examine  the  use  of  a  central  host  computer  for  distribution  and 
configuration  management  purposes. 

c.  All  of  the  systems  were  targeted  for  the  microcomputers  available  at 
USAEPG.  Because  of  the  different  configurations  in  use,  some  constraints 
exist  as  to  which  functions  can  be  used  while  still  retaining  compatibility 
with  a  majority  of  the  microcomputer  base.  Primarily  these  constraints  have 
concerned  disk  and  memory  size,  graphics  capabilities,  and  hardware 
accelerators  for  floating  point  operations.  From  a  practiced  standpoint, 
little  functionality  has  been  lost  in  conforming  to  the  minimal  configuration. 

d.  An  assessment  of  time  savings  or  inproved  quality,  due  to  the  use  of 
expert  system  aids,  can  only  be  done  qualitatively,  since  all  of  the  systems 
are  just  new  being  evaluated  using  actual  test  project  parameters.  Projected 
savings  are  considerable  in  seme  cases;  in  one  evaluation  run,  the  EVA 
assisted  in  identifying  excessive  and  unnecessary  test  requirements.  Other 
expert  systems  offer  the  potential  of  providing  preliminary  assistance  in  what 
can  be  complex  or  time  consuming  tasks.  All  of  the  systems  have  demonstrated 
the  ability  to  retain,  and  even  combine,  expertise  from  human  domain  experts. 

e.  The  present  suite  of  support  tools  all  serve  to  train  the  test 
officer  to  some  degree.  After  running  the  expert  systems  a  few  times,  the 
user  begins  to  understand  which  parameters  are  significant  for  given 
situations.  Also,  all  of  the  systems  provide  an  on-line  "help”  function  to 
inform  the  user  of  the  nature  of,  and  appropriate  response  to,  the  various 
queries  encountered.  Most  of  the  advice  offered  by  the  systems  provides  both 
the  necessary  action  and  the  reason  for  the  action;  e.g. ,  use  of  incendiary 
devices  requires  filing  a  fire  plan  with  the  post  fire  marshal. 

f.  Test  technology  for  AI  expert  systems  is  almost  nonexistent. 

However,  seme  progress  has  been  made  in  isolated  areas,  bat  much  remains  to  be 
done  before  AI  test  methodologies  can  be  considered  mature. 

1.5  ANALYSIS 

The  development  of  various  expert  systems  to  aid  the  test  officer 
demonstrates  the  usefulness  of  AI  technology.  The  systems  are  still  being 
evaluated,  and  will  probably  continue  to  evolve  to  support  more  of  the  domain 
knowledge.  Besides  the  obvious  benefits,  such  as  retained  knowledge  and 
combined  expertise  of  multiple  experts,  this  methodology  shewed  the 
feasibility  of  developing  and  using  expert  system  technology  with  existing 
microcomputer  resources.  In  addition,  inproved  productivity  and  quality  of 
work  can  be  expected  from  test  officers.  With  fewer  resources  available  to 
perform  essential  mission  functions,  productivity  and  quality  gains  may 
overshadow  other  potential  advantages  of  AI. 

The  systems  developed  for  the  investigation  addressed  individual  problem 
domains  within  the  testing  arena.  Many  of  these  domains  share  commonality  of 
information  about  test  resources,  techniques,  and  requirements  -  the 
infrastructure  of  testing.  A  broader  analysis  of  this  test  support 
infrastructure  requirements  is  appropriate.  An  early  examination  of  the 


testing  infrastructure,  with  subsequent  incorporation  of  cannon  requirements 
into  a  supporting  structure  (i.e. ,  data  bases,  networks,  geographic 
information  systems,  and  standard  information  elements) ,  could  eventually  lead 
to  an  integrated  set  of  cooperating  support  tools. 

AI  appears  in  two  complementary  areas  at  the  USAEFG:  embedded  in  test 
items  (usually.  Army  systems)  which  must  undergo  developmental  testing,  and 
used  in  test  support  systems.  The  introduction  of  AI  into  test  items  makes  it 
imperative  that  a  test  methodology  be  developed  so  USAEFG  may  perform  its 
primary  mission  of  testing.  Almost  equally  important  is  the  need  to  be  able 
to  validate  test  support  tools  which  use  AI.  Until  robust  AI  test  methods 
emerge,  the  full  potential  of  this  premising  technology  will  not  be  realized. 

1.6  CmOUSIONS 

The  investigation  was  successful  in  demonstrating  the  capability  of 
knowledge-based  systems.  This  was  accomplished  with  existing  microcomputer 
resources,  which  increased  the  availability  of  the  tools  while  minimizing 
costs.  Further  validation  of  this  microcomputer-based  expert  system 
development  methodology  over  a  complete  system  life  cycle  would  require  that 
the  prototype  tools  complete  the  ongoing  evaluation  phase.  Following  a 
favorable  evaluation,  the  tools  would  then  be  fully  developed  and  supported 
under  production  or  instrumentation  programs,  for  the  remaining  implementation 
and  maintenance  portions  of  the  life  cycle. 

Automating  the  entire  test  infrastructure  is  too  ambitious  an  effort  to 
be  absorbed  by  Sollcw-on  phases  of  this  investigation.  However,  seme 
consideration  should  be  given  to  defining  the  infrastructure  requirements  for 
the  production  version  of  knowledge-based  systems. 

Since  test  items  cure  already  being  developed  which  employ  expert  system 
technology,  and  knowledge-based  test  support  systems  have  been  shown  to  be 
beneficial,  more  emphasis  should  be  placed  upon  initiating  an  AI  test 
methodology  investigation. 

1.7  REOCMMENDATIQNS 

Further  investigation  is  recommended  in  the  following  areas: 

a.  Use  of  the  prototype  tools  should  continue  through  the  evaluation 
phase  to  attempt  to  further  validate  the  results  obtained  thus  far. 
Distribution  and  operational  considerations  associated  with  the  implementation 
phase  of  a  system  should  be  addressed,  as  well  as  maintenance  issues.  Further 
development  of  test  officer  support  tools  should  also  incorporate 
infrastructure  requirements  to  the  extent  possible. 

b.  A  separate  project  should  be  undertaken  to  analyze  the  requirements 
for  establishing  and  maintaining  an  automated  testing  infrastructure. 

c.  An  investigation  is  required  to  develop  test  procedures  for  AI.  This 
effort  would  aid  directly  in  accomplishing  the  primary  mission  of  system 
testing,  and  would  also  offer  a  means  to  validate  AI -based  test  support  tools. 
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d.  Advances  in  AI  technology  should  be  monitored  to  maintain  cognizance 
of  new  developments  in  this  rapidly  maturing  field.  This  should  include  those 
aspects  of  AI  which  have  been  explored  only  briefly  during  this  investigation. 
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SECTION  2.  DETAILS  OF  INVESTIGATION 


2.1  APPLICATION  OF  AI 

USAEPG  is  me  of  nine  test  centers  of  the  U.S.  Army's  Test  and  Evaluation 
Coranand  (TECCM) .  TECCM  has  established  two  goals  for  the  use  of  AI 
technology,  which  are  also  the  primary  goals  of  USAEPG1  s  AI  effort.  One  goal 
is  to  exploit  AI  technologies  to  enhance  the  ability  to  perform  testing.  Ihe 
other,  somewhat  obvious  goal,  is  to  test  systems  which  contain  AI.  Another 
ancillary  role  which  AI  plays  is  to  improve  upon  existing  methods  by  examining 
existing  processes,  listening  to  experts/users,  and  in  general  defining  and 
improving  the  job  to  be  done. 

USAEPG,  like  the  other  subordinate  elements  of  TECCM,  assigns  action 
officers  to  oversee  the  activities  associated  with  test  directives.  These 
test  officers  perform  a  number  of  duties.  Besides  test  planning,  the  test 
officer  is  responsible  for  monitoring  actual  test  conduct,  and  analyzing  and 
reporting  the  results.  With  test  items  increasing  in  ccnplexity  due  to  the 
increased  use  of  electronics,  computers,  and  ccrtmunications ,  the  test 
officer's  responsibilities  are  similarly  becoming  more  difficult.  This  would 
be  sufficiently  challenging  without  the  additional  burden  of  reduced  budgets 
and  increased  documentation  requirements.  At  USAEPG  alone,  approximately  100 
personnel  are  designated  as  test  officers,  with  responsibility  for  conforming 
to  all  of  the  appropriate  directives,  regulations,  and  guidelines  without 
losing  sight  of  the  primary  mission. 

The  current  phase  II  of  the  investigation  continued  earlier  efforts  to 
examine  the  potential  of  applying  microcxnputer-based  AI  technology  to  assist 
the  test  officer  (reference  2) .  Increased  enphasis  was  placed  on  managing  the 
technology  rather  than  merely  building  tools.  This  approach  required  that  all 
aspects  of  an  AI  development  infrastructure  be  addressed.  Seme  of  the 
essential  ingredients  of  this  methodology  were  the  team  organization,  training 
for  personnel  at  all  levels  in  the  organization  (including  an  apprenticeship 
program) ,  and  the  development  of  various  AI -based  support  tools  and 
exploration  of  AI  testing  issues. 

2.1.1  AI  Background. 

AI  encarpasses  a  large  and  somewhat  diverse  set  of  technologies,  ranging 
from  neural  ocnputing  to  robotics,  and  including  expert  systems,  natural 
language  processing,  and  vision  systems.  One  of  the  more  mature  technologies 
of  AI  is  that  of  expert,  or  kncwl edge-based ,  systems.  AI  developers  have 
produced  tools  known  as  expert  system  shells  that  assist  in  the  construction 
of  rule-based  expert  systems.  These  shells  allow  a  knowledge  engineer  to 
codify  logical  inferences  (rules)  about  a  given  domain,  and  to  process  the 
resulting  knowledge  base  in  order  to  provide  expertise  to  the  user. 

Most  non-trivial  expert  systems  have  been  developed  by  a  team  consisting 
of  AI  experts  and  domain  experts.  It  is  the  job  of  the  knowledge  engineer  to 
obtain  knowledge  about  a  particular  domain  through  consultation  with  one  or 
more  experts,  documented  information,  or  seme  combination  of  these  sources. 
This  knowledge  is  then  incorporated  into  an  automated  tool  which  uses  this 
expertise  in  solving  problems  within  the  domain.  Expert  system  shells  have 
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considerably  eased  the  task  of  developing  expert  system  tools,  by  providing  a 
means  to  enter  and  exercise  logical  rules  about  a  given  domain. 

Recent  developments  in  expert  system  shells  have  resulted  in  a  number  of 
tools  which  are  relatively  easy  to  use,  and  do  not  require  extensive 
programming  skills  such  as  those  normally  associated  with  using  symbolic 
programming  languages.  These  shells  have  made  it  possible  for  some  domain 
experts  to  build  expert  systems  without  assistance.  However,  knowledge 
engineering  enccrpasses  more  than  merely  entering  rules  in  the  proper  format. 

2.1.2  Application  Screening. 

Applications  proposed  for  test  officer  support  tools  were  screened  by  an 
existing  tool  which  assesses  the  probable  success  of  a  proposed  system  by 
analyzing  various  parameters  of  the  project.  This  system,  the  Expert  System 
Selector  (ES~) ,  is  itself  an  expert  system.  ES^  examines  such  factors  as  the 
availability  of  expertise  and  supporting  development  and  runtime  tools;  and 
the  suitability  and  feasibility  of  an  expert  system  solution.  It  then 
provides  a  qualitative  score  of  the  overall  success  potential.  Proposed 
concepts  had  to  be  sufficiently  well  defined  to  allow  grading  by  the  ES  . 

Only  then  could  those  concepts  be  considered  for  development.  This  approach, 
in  fact,  was  used  to  screen  ideas  for  the  workshops,  and  was  responsible  for 
the  elimination  of  what  would  have  been  poorly  suited  or  overly  ambitious 
suggestions. 

2.1.3  MicroocTOJUter  Development  Environment. 

The  computing  resources  of  USAEPG  include  a  variety  of  mainframe,  mini, 
micro,  and  special-purpose  AI  computers.  However,  only  the  ubiquitous 
microcomputer  is  readily  available  to  the  test  officer  for  planning  functions. 
Earlier  AI  efforts  demonstrated  the  practicality  of  AI  systems  targeted  for 
these  machines,  although  micrcconputer  implementations  are  not  without  their 
own  unique  challenges.  One  problem  centers  on  the  need  for  practical  methods 
to  handle  distribution  and  configuration  management.  Another  problem,  not 
strictly  limited  to  applications  on  the  small  machines,  is  the  need  for 
production  level  systems  to  access  informatics!  and  knowledge  on  the  testing 
infrastructure. 

Automatics!  of  the  testing  infrastructure  within  the  context  of  a  large 
organization  requires  at  least  two  types  of  knowledge.  The  first  type, 
knowledge  of  the  domain  in  which  the  system  is  to  advise  and  assist,  is  termed 
domain  expertise,  and  is  the  object  of  the  knowledge  acquisition  effort  as 
commonly  described  in  AI  literature.  The  second  type  involves  information 
concerning  the  administrative,  organizational,  and  regulatory  environment 
within  which  the  expert  and  system  must  operate.  Within  USAEPG,  as  with  most 
organizations,  requisite  information  is  widely  available,  but  from  a  variety 
of  sources.  At  this  time,  there  is  no  central  point  for  maintenance  of  or 
access  to  this  infrastructure  information. 

2.1.4  Team  Structure. 


USAEPG  AI  efforts  are  managed  out  of  an  office  in  the  Software  and 
Interoperability  Division.  The  team  consists  of  management,  engineer,  and 
apprenticeship  personnel,  supplemented  with  personnel  from  an  existing 
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technical  support  ocntract.  Upper  management  also  plays  a  key  role  in 
obtaining  the  ocnmitment  and  resources  so  essential  to  the  insertion  of  new 
technology.  Because  a  successful  technology  development  program  requires  both 
adequate  tools  and  the  management  and  technical  skills  to  effectively  use  the 
technology,  a  considerable  amount  of  esrphasis  is  placed  on  training  at  all 
levels  in  the  organization. 

2.1.5  Management  Involvement. 

Management  participation  is  an  essential  element  of  any  new  technology 
insertion  effort.  The  approach  followed  during  this  investigation  included 
aspects  of  training  as  well  as  oversight  activities.  Management  training  was 
obtained  through  special  courses  and  by  participation  in  workshop  activities. 
The  Management  of  Expert  Systems  Course,  presented  on  site  by  the  U.S.  Army 
Signal  Center  and  School,  was  attended  by  15  USAEPG  and  7  TECCM  management 
personnel.  Another  7  personnel  from  other  organizations  at  Fort  Huachuca  also 
attended.  The  course  covered  all  aspects  of  expert  systems  from  planning  to 
actual  design-on-paper  of  a  class  exercise.  This  coupled  with  active 
participation  in  AI  development  efforts  gave  management  the  insight  to 
properly  incorporate  this  new  technology. 

Management  oversight  was  cultivated  through  the  establishment  of  a 
steering  ccmmittee.  The  goals  of  the  committee  meetings  are  to  provide  the 
communication  channel  to  senior  management  from  the  AI  cell  and  provide  a 
forum  for  resource  ccrmitments  and  priorities  to  be  assigned  to  proposed 
projects,  based  on  ocrmand  perspectives. 

2.1.6  Apprenticeship  Program. 

An  apprenticeship  program  is  used  to  train  new  members  of  the  AI  team,  as 
well  as  personnel  outside  the  division.  For  team  members,  the  apprenticeship 
represents  part  of  the  initial  training  which  is  received.  For  others,  the 
apprenticeship  is  a  way  to  develop  projects  quickly. 

An  apprentice  begins  by  being  temporarily  assigned  full  time  to  the  AI 
Office,  minimizing  the  interruptions  which  would  occur  if  he  or  she  were  to 
remain  in  their  regular  assignment.  Although  the  actual  period  of  training 
varies  with  individual  ability  and  desired  acccnpl ishments ,  the  average  time 
allowed  is  fcur  months.  At  the  end  of  this  period  the  apprentice  will  be 
familiar  with  the  basics  of  developing  rule-based  expert  systems  with  the  use 
of  shell  tools.  Also,  the  trainee  will  have  developed  at  least  one  prototype 
application  to  satisfy  same  need  at  their  heme  office. 

The  apprenticeship  begins  by  attending  a  two  week  course  in  basic  expert 
system  building  and  participating  in  local  workshops.  While  this  training  is 
generally  available  to  most  personnel,  the  apprenticeship  offers  a  number  of 
advantages.  Most  people  who  attend  long  courses  on  their  cwn  return 
immediately  to  their  heme  office  and  spend  their  time  trying  to  catch  up  on 
work  they  missed.  By  the  time  they  get  around  to  applying  the  techniques  they 
learned  in  the  AI  course,  much  of  the  effectiveness  of  the  training  will  have 
been  lost.  In  the  apprenticeship  program,  students  are  able  to  learn  new 
concepts  and  tools  and  immediately  begin  to  apply  this  knowledge.  Not  only 
does  this  greatly  improve  the  education  process,  but  it  allows  more  advanced 
techniques  to  be  assimilated  within  a  shorter  time.  Augmenting  the  basic 
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training  by  exposure  and  actual  experience  with  concepts  merely  touched  upon 
in  the  basic  courses  allows  the  apprentice  to  build  better  systems  more 
effectively  when  they  return  to  their  hone  office. 

While  the  apprentice  and  his  or  her  heme  office  serve  to  benefit  directly 
from  the  program,  the  AI  office  is  also  ccnpensated.  One  of  the  goals  of  the 
AI  office  is  to  educate  as  many  personnel  as  possible  on  the  benefits  and 
capabilities  of  AI.  Apprentices  help  achieve  this  goal  by  serving  as  tutors 
of  AI  to  members  of  their  heme  office.  Also,  while  an  apprentice,  a  person 
will  usually  be  assigned  to  participate  in  the  development  of  an  expert  system 
or  expert  system  tools  which  support  current  efforts  of  the  AI  team.  The 
synergism  provided  by  this  program  makes  this  a  good  approach  for  leveraging 
the  limited  resources  of  an  organization. 

2.1.7  In-house  Workshops. 

In-house  workshops  provided  much  of  the  training  on  specific  rule-based 
tools.  (Additional  tool  training  was  obtained  from  the  U.S.  Army  Signal 
Center  and  School  for  the  M.l  shell.)  Attendees  included  USAEPG  AI  team 
members,  AI  developers  from  other  activities,  and,  in  sane  instances,  test 
experts.  The  objective  of  the  workshops  was  to  familiarize  personnel  with  the 
technology  and  to  solicit  ideas  for  further  development.  Workshops  consisted 
of  approximately  ten  students  (or  student/expert  teams) ,  each  of  wham  built  a 
small  expert  system  as  a  class  exercise.  Of  these,  some  ideas  were  selected 
for  development  of  a  prototype  system,  based  on  a  management  review  of  the 
class  projects.  One  side  benefit  was  the  exposure  of  both  management  and  test 
experts  to  the  capabilities  and  limitations  of  expert  systems. 

In  addition  to  training  in  rule-based  shells,  the  workshops  have  provided 
a  forum  for  exposure  to  other  aspects  of  the  technology.  Other  areas  touched 
upon  have  been  neural  networks,  hypertext,  and  exairple-based  shells. 

2.1.8  TEOCM  Involvement. 

TEXXM,  parent  command  of  USAEPG  and  other  test  centers,  has  supported  AI 
technology  insertion  efforts  in  a  number  of  ways.  USAEPG  has  been  designated 
as  the  support  oenter  for  AI  within  the  caimand,  providing  planning  functions 
and  training  such  as  the  workshops.  Workshops  are  often  held  in  conjunction 
with  AI  planning  meetings,  so  that  AI  contacts  from  the  other  test  centers  can 
participate  in  both  functions.  TEOCM  has  also  designated  the  chief  of  the 
USAEPG  AI  Office  to  act  as  technical  agent  for  AI  matters  within  TEOCM,  and  to 
represent  the  other  test  centers.  This  considerable  commitment  on  the  part  of 
TEOCM  to  share  resources  has  helped  leverage  the  limited  assets  of  the 
individual  test  centers. 
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2.2  AT  APPLICATION  DFVRTOWFT7T 

An  AI  expert  system  development  methodology  was  synthesized  from  the 
lessons  learned  firm  previous  projects,  AI  technology  capabilities,  ccpputer 
resource  availability,  elements  of  the  testing  infrastructure.  This  resulted 
in  an  approach  similar  to  that  used  by  industry  for  smaller  AI  applications: 

a.  Acquisition  of  microcomputer  development  tools  and  development  of 
related  personnel  skills. 

b.  Identification  of  suitable  applications. 

c.  Teaming  of  a  knowledge  engineer  and  domain  expert (s) . 

d.  Prototyping  and  iterative  development  of  the  expert  systems. 

The  result  of  implementing  this  methodology  was  the  generation  of  a 
number  of  small  expert  systems  which  address  problems  encountered  by  the  test 
officer.  Most  of  the  systems  deal  with  requirements  during  the  planning 
phases  of  a  test.  This  is  not  an  indication  that  expert  systems  are  not 
suitable  for  test  conduct  or  reporting  activities,  but  probably  does  reflect 
the  greater  stability  and  better  defined  nature  of  the  planning  stage.  That 
is,  test  plans  and  environmental  documentation  are  always  required,  regardless 
of  other  variations  in  the  test  conduct  activity.  Another  drawback  to 
addressing  test  conduct  requirements  is  that  these  applications  are  relatively 
large,  and  would  consume  all  of  the  available  resources  for  a  single  system. 

2.2.1  New  Expert  Systems. 

The  prototype  test  officer  support  tools  built  during  the  investigation 
are  described  belcw.  For  each  system,  the  purpose  and  goals,  domain, 
requirements,  description,  design  characteristics,  benefits,  and  status  are 
briefly  described. 

2.2. 1.1  Contract  Performance  Evaluation  -  Advisor. 

2. 2. 1.1.1  Purpose/Goals.  The  Contract:  Performance  Evaluation  -  Advisor 
(CPEA)  has  been  developed  to  assist  test  officers  in  evaluating  a  contractor's 
performance  of  work.  The  goal  of  CPEA  is  to  ensure  that  uniform  standards  are 
applied  to  performance  evaluations  for  all  of  the  tasking  on  large  contracts. 

2.2. 1.1.2  Dcmain/Expertise.  Kncwledge  used  in  the  CPEA  system  was  acquired 
from  experienced  test  officers  and  contract:  support  documentation  such  as 
standard  operating  procedures  (SOPs) .  Test  officers  also  played  a  major  role 
in  the  systems  design,  which  was  developed  by  personnel  in  the  apprenticeship 
program.  (However,  prior  to  the  apprenticeship  the  developer  had  served  in 
the  division  in  a  capacity  that  required  him  to  perform  contract  evaluations.) 

2.2. 1.1. 3  Requirements.  The  requirement  for  a  contract  evaluation  expert 
system  came  about  as  a  result  of  the  need  to  standardize  the  process  in  which 
scoring  is  done.  Also,  when  management  selected  the  CPEA  as  a  workshop  system 
to  be  developed  into  prototype  form,  they  were  interested  in  speeding  up  the 
process.  Other  requirements  included  automating  the  evaluation  process, 
generating  reports,  designing  the  program  so  users  would  be  forced  to  address 
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each  evaluation  subfactor,  and  addressing  situations  where  the  answer  to  a 
particular  question  in  unknown.  Another  desired  feature  was  to  allow  the  test 
officer  control  over  the  final  score  received  by  a  sub- factor;  the  test 
officer  is  given  a  five  point  range  of  scores  and  is  allowed  to  make  the  final 
decision  on  a  score  taken  from  within  the  range. 

2. 2. 1.1. 4  Description.  CFEA  asks  the  user  specific,  low  level  questions  for 
which  they  are  likely  to  knew  the  answers.  Based  on  the  responses,  a 
recommended  range  of  scores  are  offered  to  the  test  officer  for  each  subfactor 
in  the  evaluation.  After  all  of  the  questions  (56  in  number)  are  answered, 
CPEA  prints  out  a  report,  and  saves  the  information  on  disk  storage.  There  is 
also  an  option  available  for  printing  reports  that  depict  the  answer  to  each 
question. 


2. 2. 1.1. 5  Desicm/Develocment  characteristics. 

The  system  was  developed  on  an  MS-DOS  compatible  microcomputer.  A 
rule-based  expert  system  shell,  M.l,  by  Tekncwledge,  was  used  to  provide  the 
inferencing  environment  in  which  the  CFEA  knowledge  base  is  run. 

The  system  inferencing  is  dene  using  backwards  chaining.  There  are  three 
levels  of  inferencing  that  work  with  71  rules.  Ccnclusions/recommendations 
are  made  using  weighted  scores  and  fact  tables  that  relate  responses  to 
questions.  Combined  responses  for  a  subfactor  result  in  a  five  point 
recommended  range.  The  test  officer  can  then  select  the  number  within  the 
range  which  is  most  appropriate. 

2. 2. 1.1. 6  Validation/Test  Methodology.  CPEA  was  tested  by  manually  scoring 
the  same  sub-factors  as  used  in  the  CFEA  rules  and  fact  tables.  The  output  of 
this  method  was  then  evaluated  to  insure  that  answers  were  the  same  as  those 
produced  by  the  automated  expert.  The  program  is  still  in  the  process  of 
being  validated.  Validation  also  is  being  performed  by  comparing  a  test 
officer's  scores  generated  manually  with  scores  generated  using  CPEA. 

2. 2. 1.1. 7  Benefit/Use.  CFEA  has  acocrplished  the  goal  of  standardizing 
scoring  procedures.  However,  it  is  net.  likely  to  save  time  relative  to  manual 
methods,  nor  to  result  in  significant  changes  to  the  contractor's  award  fee. 

2. 2. 1.1. 8  Development  Status. 

The  system  is  in  its  second  phase  of  development.  This  phase  included 
upgrades  in  the  area  of  default  scores,  and  special  reports  to  be  used  by 
management  to  review  a  test  officers  evaluation. 

Future  development  will  consist  of  interfacing  CPEA  to  a  local  area 
network  that  will  then  interface  with  Lotus  1-2-3  and  dBASE  files. 

2.2. 1.2  Security  System. 

2.2. 1.2.1  Purpose/Goals.  The  Security  Expert  System  (SEC)  was  built  with  one 
underlying  theme,  enforcement  of  the  security  SOPs  of  the  USAEFG.  The  main 
goal  of  the  system  was  a  combination  of  the  following  ideas:  1)  Make  all  test 
officers  aware  of  security  needs  for  a  test.  2)  Give  the  test  officer  a  list 
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of  security  do's  and  don' ts.  3)  Help  the  Intelligence  &  Security  Division  do 
security  risk  analyses  of  the  testing  programs  at  USAEPG. 

2. 2. 1.2. 2  Dcmain/Expertise.  Expertise  for  the  development  of  SEC  came  from 
the  automated  data  processing  (ADP)  security  manager,  the  information  security 
specialist  of  Intelligence  &  Security  Division,  Army  Regulation  380-5,  and  DA 
Pamphlet  190-51. 

2.2. 1.2.3  Requirements.  The  SEC  is  required  to  determine  if  a  test  project 
needs  ary  hard  products  (e.g. ,  guards,  safes,  or  courier  orders)  or  advice 
(for  example,  procedures  to  be  followed)  that  Intelligence  &  Security  Division 
can  provide.  The  system  also  must  aid  the  user  in  doing  a  security  risk 
analysis  of  the  test  site  and  project.  Once  completed,  the  risk  analysis  will 
identify  a  set  of  security  procedures  and  protective  measures  that  the  test 
officer  is  responsible  for. 

2. 2. 1.2. 4  Description.  The  SEC  consists  of  three  main  modules,  covering  the 
areas  of  information  security  and  physical  security.  As  each  module  is 
completed  the  responses  the  user  gave  are  stored  in  ASCII  files  for  later 
review  and  print  out.  The  first  module  collects  the  background  information  on 
the  test  project  (e.g. ,  project  title,  TECJCM  project  number,  the  test 
officer's  name  and  testing  dates) .  Next,  this  module  covers  information 
security  issues  and  procedures,  indicating  how  to  secure  hard  information. 
Another  module  then  does  a  risk  analysis  to  identify  the  security  procedures 
and  protective  measures  that  the  test  officer  must  be  aware  of  as  they  apply 
to  aircraft,  vehicles,  and  electronic  equipment.  The  last  module  collects  all 
the  saved  responses  and  prints  out  a  report  for  the  Intelligence  &  Security 
Division  files. 

2. 2. 1.2. 5  Desicm/Develcanent  characteristics.  SEC  is  a  microccmputer-based 
expert  system  written  using  the  M.l  shell  and  a  text  editor.  The  expert 
system  was  divided  into  several  procedural  modules.  These  modules  were 
designed  to  be  individually  called  into  memory  and  run.  Then  information 
learned  is  saved  to  separate  files.  The  system  was  built  in  pieces, 
individual  modules  being  set  up  to  handle  small  requirements  of  the  security 
problem  (e.g. ,  conducting  a  risk  analysis,  or  determining  if  guards  are 
required  at  a  test  site) . 

2.2. 1.2.6  Validation/Test  Methodology.  As  the  system  was  developed  and 
modules  added,  a  security  specialist  frcm  Intelligence  &  Security  Division  ran 
the  system  to  evaluate  the  latest  changes.  A  snail  group  of  test  officers 
acted  as  a  beta  test  group  to  further  evaluate  and  improve  the  system. 

2. 2. 1.2. 7  Benefits/Use.  SEC  was  designed  for  use  by  test  officers  as  an  aid 
for  security  matters  as  they  pertain  to  a  test  project.  It  will  be  their 
first  and  most  readily  available  source  of  security  guidance.  There  is  no 
requirement  at  the  USAEPG  for  test  officers  to  develop  a  security  plan  for 
their  test,  but  they  are  responsible  for  the  security  of  the  test  site, 
equipment,  and  data.  This  expert  system  will  help  them  fulfill  this 
responsibil ity . 

2. 2. 1.2. 8  Development  Status.  The  prototype  SEC  is  ready  for  distribution 
and  use  thrcughout  USAEPG.  Presently,  no  user  manual  or  documentation  exists, 
yet  these  are  not  needed  to  load  or  run  the  system  on  a  microcomputer. 
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Intelligence  &  Security  Division  plans  to  introduce  the  system  to  all  of 
USAEPG  and  via  a  ocnmand  directive  require  its  use  for  all  test  programs  in 
the  future.  Training  will  be  accomplished  through  formal  classes,  with 
student  cctrmerrts  used  to  formulate  requirements  for  future  enhancements. 

2.2.2  Previous  Applications. 

Previous  phases  of  the  investigation  resulted  in  the  development  of  a 
number  of  applications.  Those  systems  described  in  earlier  reports  are 
included  in  Appendix  D.  Currently  these  systems  are  in  an  evaluation  phase  or 
are  in  the  process  of  entering  a  production  version  phase  under  other  funding 
programs. 

One  of  the  benefits  of  AI  systems  is  their  use  to  retain  expertise. 

Since  the  development  of  these  earlier  prototypes,  the  value  of  this  benefit 
has  been  validated  by  the  personnel  changeover  of  experts  for  two  of  the 
applications.  Another  benefit  is  cost  savings  over  existing  methods.  While 
difficult  to  quantify,  especially  for  systems  in  the  prototype  stage,  seme 
savings  have  already  been  realized.  During  one  use  of  the  EVA  environmental 
screening  system,  an  anticipated  need  for  fifty  support  vehicles  was  reduced 
to  the  real  requirement  of  three  vehicles.  This  reduction  to  appropriate 
levels  resulted  in  direct  cost  savings  and  a  test  which  was  more  benign  to  the 
environment. 

2.2.3  Other  Applications. 

The  systems  described  in  this  section  were  written  to  explore  potential 
uses  and  capabilities  of  AI,  and  as  part  of  the  apprenticeship  training 
exercises.  Being  exploratory  in  nature,  these  ideas  were  not  subject  to  the 
Es  screening  process  used  on  other  applications.  These  applications  required 
approximately  four  manweeks  each  to  develop  into  a  demonstration  form.  One  of 
the  objectives  was  to  examine  the  use  of  expert  system  shells  to  satisfy 
conventional  application  requirements  (for  exanple,  shells  provide  a  built  in 
capability  for  queries  and  report  generation) . 

2.2. 3.1  Budget  Spreadsheet  Analysis  Aid. 

2. 2. 3. 1.1  Description.  BLJD2 ,  Budget  Spreadsheet  Analysis  Aid,  is  a  small 
rule-based  expert  system  written  largely  in  lotus  1-2-3  macros  and  the  expert 
system  shell,  EXSYS.  It  was  designed  to  help  the  Directorate  Chief  by  giving 
him  a  quick  analysis  of  a  budgetary  spreadsheet.  The  system  does  a  quick  scan 
of  the  monthly  Obligation  and  Disbursement  spreadsheet  used  here  at  USAEPG. 
This  lotus  spreadsheet  contains  a  list  of  test  projects  or  project  funds, 
indicating  a  past  history  and  current  levels  of  disbursed  and  obligated  funds. 
BUD2  reads  the  output  frem  Lotus  and  flags  projects  that  are  off  track  with 
the  planned  obligation  and  disbursement  levels  for  that  project. 

2. 2. 3. 1.2  Lessons  Learned. 

lotus  macros  can  be  a  pewerful  progranming  language.  Because  of  BUD2's 
size  and  simplicity,  all  the  rules  written  in  the  EXSYS  shell  could  have  been 
done  entirely  in  lotus  macro  code.  Executing  solely  in  lotus  would  reduce  the 
operating  time  for  the  user. 
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Within  USAEFG,  the  users  of  application  software,  like  Lotus  1-2-3 ,  have 
their  own  set-up  or  configuration  for  that  software.  EtJD2  was  designed  to  run 
the  Lotus  software  package  an  the  user's  machine.  To  get  around  differences 
in  the  set-up,  BUD2  must  first  reset  several  parameters  to  the  same  values 
used  on  the  developer's  cctrputer. 

2. 2. 3. 2  Tape  Test  Expert  System. 

2. 2. 3. 2.1  Description.  The  Tape  Test  Expert  System  (TIES),  developed  with 
M.l,  is  designed  to  assist  small  Army  units  in  the  administrative  task  of 
determining  a  soldier's  body  fat  percentage.  The  system  is  used  when  soldiers 
fail  to  meet  Army  screening  weight  requirements  as  identified  in  AR  600-9. 

TIES  requests  measurements  taken  from  certain  areas  of  the  body.  That  data  is 
then  utilized  to  look  up  corresponding  figures  and  to  perform  seme  elementary 
math.  The  results  produced  by  TIES  shows  a  soldier's  body  fat  percentage  and 
tells  whether  or  not  the  soldier  meets  the  Army's  requirements.  TIES  is  not 
an  expert  system.  It  is  simply  an  automated  way  of  performing  this  particular 
function  quicker  and  more  efficiently. 

2. 2. 3. 2. 2  Lessons  Learned.  TIES  has  provided  users  a  more  consistent  method 
of  performing  body  fat  analysis  procedures  and  calculations.  TIES  also 
reduces  the  amount  of  stress  on  the  soldiers  being  examined,  by  providing 
results  within  thirty  seconds  after  data  is  collected.  TIES  is  currently 
being  used  consistently  by  cane  USAEPG  oonpany.  Due  to  the  fact  that  there  are 
several  programs  available  that  can  perform  this  function,  no  future 
development  is  planned. 

2. 2. 3. 3  h i.1  Look— up . 

2. 2. 3. 3.1  Description .  Physical  Training  (FT)  Look-up  is  a  prototype  system 
for  a  microcomputer,  written  using  the  expert  system  shell  M.l,  and  Sidekick 
as  the  text  editor.  The  knowledge  used  in  this  system  came  from  the  Army 
Physical  Fitness  Test  scoring  card,  DA  Form  705  FT.  It  was  developed  at  the 
request  of  operations  personnel,  ft  look-up  is  a  knowledge-based  program, 
intended  to  help  the  records  clerk  in  the  cotpany  training  office  fill  out  a 
DA  705  scoring  card  after  a  soldier  has  taken  the  Army  Physical  Fitness  Test. 
The  clerk  first  enters  the  soldier's  name,  age,  and  sex.  FT  Look-up  then  asks 
for  the  number  of  repetitions  of  push-ups  and  sit-ups  the  soldier  completed . 

If  the  numbers  are  valid,  the  expert  system  returns  the  point  value 
corresponding  to  that  number  of  repetitions .  The  user  then  records  these 
numbers  on  the  score  card.  The  system  then  asks  for  the  time  it  took  the 
soldier  to  ccnplete  the  two-mile  run.  Once  that  information  is  entered,  the 
system  returns  the  corresponding  point  value  for  that  time  and  the  sum  of  the 
three  point  values  for  a  total  FT  test  score.  The  last  two  numbers  are  then 
recorded  and  the  card  is  ccnplete  without  the  clerk  having  to  search  the 
scoring  tables  for  a  point  value,  add  the  values,  and  double  check  the  point 
values  taken  frem  the  table. 

2. 2. 3. 3. 2  Lessons  Learned. 

This  system  exenplifies  the  use  of  an  AI  shell  in  the  programing 
language  role.  FT  Look-up  doesn't  contain  any  rules  from  an  expert,  but 
rather  is  a  lot  of  simple  procedures  linked  together  to  act  like  semeone  who 
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is  filling  out  a  DA  705  PT.  The  procedures  are  actually  in  the  body  of 
if-then  rules. 

FT  Look-up  has  turned  out  to  be  one  of  the  most  regularly  used  systems  in 
USAEPG,  for  the  following  reasons: 

a.  The  intended  user  defined  the  product  and  stated  the  requirements. 

b.  FT  Look-up  saves  the  user  time  in  doing  a  tedious  job. 

c.  The  records  clerk  has  the  obligation,  by  carmand  directive,  to  keep 
track  of  DA  705  cards  for  every  soldier  in  the  unit. 

2.2.4  Workshop  Systems. 

2.2. 4.1  Workshop  Overview.  One  of  the  goals  of  the  USAEPG  AI  Office  is  to 
educate  both  users  and  management  on  the  capabilities  of  AI  technology.  One 
method  of  achieving  this  goal  has  been  to  conduct  periodic  workshops.  In  the 
workshops,  attendees  learn  about  the  basic  skills  used  to  build  expert  systems 
(e.g. ,  knowledge  acquisition  -  the  systematic  analysis  and  logical 
formalization  of  domain  knowledge)  and  become  familiar  with  the  use  of  an 
expert  system  development  tool.  To  enhance  the  practicality  of  the  course, 
students  were  asked  to  select  an  actual  problem  from  their  workplace  for  which 
they  themselves  could  provide  the  expertise.  These  ideas  were  then 
implemented  during  the  course  of  the  workshop  both  as  a  learning  aid  and  as  an 
initial  prototype  for  possible  further  development.  At  the  end  of  the 
workshop,  the  prototypes  were  demonstrated  to  management  as  exanples  of  the 
types  of  real  problems  which  can  be  readily  solved  with  the  technology. 

2. 2. 4. 2  Workshop  Applications.  Applications  developed  during  the  workshop 
were  limited  in  scope  due  to  the  fact  that  time  permitted  only  the  development 
of  demonstration  versions.  Hcwever,  seme  ideas,  such  as  contract  performance 
evaluation,  were  selected  by  management  for  further  prototype  development. 

This  development  resulted  in  the  CFEA  system  described  earlier.  Other  good 
application  areas  for  workshop  exercises  are  classification,  selection,  and 
policy  enforcement. 

2.3  NEW  TECHNOLOGY 

Several  activities  supported  the  exploitation  of  new  technology  in  the  AI 
field.  One  effort  was  to  develop  a  hypertext  document  handling  tool.  Other 
efforts  in  this  area  were  an  examination  of  example-based  AI  development 
shells,  and  a  rule-based  shell  recently  released  by  the  National  Aeronautics 
and  Space  Administration  (NASA) . 

2.3.1  DocuView  Hypertext  Tool. 

2. 3. 1.1  Purpose /Goals.  The  intended  use  of  DocuView  is  for  displaying 
general  textual  information  on  a  computer  screen.  Hypertext  expansion 
techniques  are  used  for  highlighting  certain  phrases  within  a  document. 

Through  selection  of  these  phrases,  those  techniques  allow  a  nonlinear 
traversed  of  the  document. 
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2. 3. 1.2  Damain/Expertise.  This  software  program  is  capable  of  being  used 
wherever  documents  or  general  text  materials  need  to  be  separated  into  pages 
for  display  purposes.  The  document  developer,  throogh  various  contends 
embedded  in  the  text,  has  the  flexibility  to  present  the  information  in  the 
most  suitable  manner  for  the  particular  domain  being  handled.  The  user,  based 
on  actual  needs  during  presentation,  has  the  control  to  dynamically  alter  the 
order  in  which  the  material  is  viewed.  Thus,  both  the  developer  and  laser  play 
a  key  role  in  the  assimilation  of  hypertext  information. 

2. 3. 1.3  Requirements.  This  type  of  software  tool  is  needed  so  that  documents 
residing  on  the  computer  can  be  broken  into  logically  defined  pages  for 
presentation  as  windows  on  a  computer  display  screen.  Documents  must  be 
stored  in  a  form  which  allows  modification  by  most  text  processors,  yet  must 
also  be  directly  presentable  by  the  document  viewing  tool. 

2. 3. 1.4  Description.  The  DocuView  tool  is  a  software  package  consisting  of  a 
main  program  and  numerous  subprograms  and  functions  written  in  a  conventional 
computer  language.  The  software  is  designed  to  present  the  contents  of  a 
document  file,  referred  to  as  an  object  file,  on  a  microcomputer  display 
screen  in  user  specified  pages.  Each  page  is  defined  to  have  its  own  window 
at  a  chosen  location  on  the  screen,  and  has  a  set  of  parameters  which  specify 
color,  size,  and  other  options.  Pages  are  inserted  into  a  text  file  by  the 
addition  of  DocuView  command  lines.  Other  command  lines  are  entered  into  the 
text  to  signify  selected  states  or  state  changes.  These  commands  make  it 
possible  for  the  DocuView  user  to  work  with  varying  document  types  and 
contents  without  experiencing  conflicts  between  contends  and  the  textual 
contents.  For  example,  the  command  words  and  delimiters  used  in  the  text  are 
changeable  as  needed  by  the  user.  As  an  example,  the  exclamation  point 
character  used  to  delimit  highlighted  phrases  can  be  changed  to  seme  other 
character  when  conflicts  in  the  document  text  arise. 

2. 3. 1.5  Design/ Development  Characteristics.  The  most  significant  feature  of 
DocuView  is  that  it  allows  the  document  being  analyzed  to  be  broken  up  into 
pages  for  presentation  on  a  computer  display  screen  and  that  on  these  pages  of 
text,  chosen  phrases  can  be  highlighted  for  hypertext  expansion  into  yet 
additional  pages  of  commentary  or  description.  The  display  of  pages  and 
hypertext  expansion  of  selected  phrases  can  be  dene  recursively  for  page  after 
page  of  textual  information. 

2. 3. 1.6  Benefits/Use.  Information  new  stored  on  computer  media  or  available 
in  such  form  can  be  conveniently  displayed  on  a  computer  screen.  No 
significant  changes  to  the  original  document  are  necessary.  At  the  same  time 
any  text  processing  of  the  document  is  readily  possible  with  conventional  text 
editors.  The  real  benefits  though  are  to  be  realized  with  the  display  of  only 
pertinent  information  and  the  resulting  improvement  in  assimilating  new 
information. 

2. 3. 1.7  Development  Status.  Development  has  reached  a  stage  where  an  initial 
version  of  the  DocuView  tool  was  distributed  at  one  of  the  AI  workshops  and  to 
other  interested  parties.  Presently,  comments  received  frxxn  formal  and 
informal  reviews  are  being  incorporated  into  a  new  version.  Possible 
improvements  include  increasing  the  types  of  parameters  which  are  user- 
defined,  providing  a  selective  print  function,  and  allowing  easier  use  by  such 
features  as  automatic  sizing  of  text  to  fit  a  windew. 


2-11 


2.3.2 


Example-based  shells  use  an  inductive  inference  methodology.  This 
technique  accepts  objects  of  a  known  class  (i.e. ,  the  "exanples")  with  a  fixed 
collection  of  attributes.  The  attributes  are  distinguishing  characteristics 
which  determine  which  set  of  objects  (clasi)  a  given  object  belongs  to.  After 
processing  by  an  inductive  algorithm,  a  decision  tree  with  attributes  is 
produced  which  may  then  be  used  to  classify  unknown  objects.  This  methodology 
is  well  suited  for  classification,  obviously,  and  diagnostic  problems. 

Example-based  tools  can  be  extremely  easy  to  use  since  the  development 
environment  builds  the  rules  automatically,  given  example  situations  as  input. 
Although  the  tools  are  easy  to  use,  sane  caution  must  be  exercised  to  ensure 
that  a  potential  problem  domain  is  amenable  to  use  of  inductive  techniques. 
Examples,  of  course,  must  exist,  or  the  domain  expert  must  be  able  to  provide 
examples.  Less  obvious  is  that  attributes  which  distinguish  one  set  of 
objects  from  another  must  be  defined.  And,  the  examples  used  to  build  a 
system  must  be  representative  of  the  domain,  and  must  cover  all  of  the  clasr.es 
among  which  the  system  must  distinguish.  A  good  design  methodology  would  also 
provide  for  ordering  the  attributes  by  the  cost  of  obtaining  the  information. 
(An  excellent  paradigm  of  this  last  requirement  is  offered  by  a  hypothetical 
medical  diagnostic  system.  Attributes  such  as  temperature,  pulse  rate,  etc. 
would  be  used  to  identify  a  pathological  condition,  if  at  all  possible,  prior 
to  requiring  exploratory  surgery.) 

Example-based  shells  can  provide  other  ’  _  .^itr  as  well.  Some  are  good 
for  discovering  any  underlying  structure  1.1  lew  level  data  (i.e.,  they  perform 
a  "factor"  or  attribute  analysis  Ter  the  developer) .  Another  useful  feature 
of  some  shells  is  the  ability  to  provide  counterexanples  where  examples  are 
are  either  too  few  or  much  too  extensive  (e.g. ,  a  medical  diagnostic  system 
which  attempted  to  describe  all  the  attributes,  of  a  well  person) . 

To  assess  the  potential  of  example-based  tools  an  existing  expert  system, 
the  Software  Analyst's  Assistant  (SAA) ,  was  converted  from  a  minicomputer  to  a 
microcxmputer  environment  using  the  lst-CLASS  shell.  The  SAA  was  used  in  this 
capacity  because  the  rules  had  been  developed  as  exanples,  which  made  the 
conversion  process  itself  a  trivial  undertaking,  even  though  the  SAA  is  a 
medium-sized  system  ( approximately  500  rules) .  This  exercise  provided  much 
insight  into  the  specific  features  of  the  lst-CLASS  tool.  (These  will  not  be 
described  in  detail,  other  than  to  mention  that  the  tool  offers  considerable 
flexibility,  an  extremely  user-friendly  interface,  and  a  classification 
algorithm  with  a  linear  time  function. )  Most  important  is  the  capability  to 
graphically  display  the  decision  tree  built  by  the  inductive  algorithm.  This 
logic  tree  can  be  examined  to  avoid  creation  of  extraneous  inferences,  and 
conversely,  to  identify  situations  unaccounted  for.  (The  conversion  of  the 
SAA  resulted  in  the  discovery  of  one  instance  of  the  latter  case,  although  the 
impact  of  this  oversight  in  SAA  operation  turned  out  to  be  insignificant.) 

Used  properly,  with  appropriately  structured  problems,  an  example-based 
shell  can  be  a  tremendously  effective  development  tool.  Microcomputer 
versions  can  adequately  handle  read  size  problem  domains  with  performance 
comparable  to,  or  better  than,  rule-based  shells.  But  the  most  interesting 
feature  (at  least  for  a  testing  organization)  is  the  ability  to  validate  a 
system  by  visual  and  automatic  examination  of  the  logic  tree. 
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2.3.3  NASA  CUPS. 


The  NASA  C  language  Integrated  Production  System  (CLIPS)  tool  is  a 
development  and  delivery  expert  system  tool  which  provides  a  ccrplete 
environment  for  the  construction  of  rule-based  expert  systems.  (Tools  such  as 
M.l  require  the  user  to  provide  his/her  own  text  editor.)  Versions  are 
available  for  a  number  of  ccrputer  environments,  including  a  microcomputer 
environment  which  is  compatible  with  USAEFG  resources.  The  CLIPS  distribution 
package  has  a  number  of  potentially  useful  and  unique  features.  Source  code 
and  documentation  are  available  at  no  cost  to  government  agencies  and  their 
contractors  (call  the  CLIPS  Help  Line  at  (713)  280-2233).  The  system  is 
currently  limited  to  forward  chaining;  but  it  has  a  powerful  rule  syntax,  is 
portable,  can  be  embedded  within  conventional  procedural  code,  and  can  be 
extended  with  the  addition  of  user-defined  functions.  In  addition,  CLIPS 
comes  with  a  utility  to  aid  in  verification  and  validation  of  rules  by 
providing  cross  referencing  of  fact  relations,  style  checking,  and  semantic 
error  checking.  An  Ada  version  of  CLIPS  (current  implementation  is  in  C)  is 
also  being  developed. 

Experience  with  the  CLIPS  environment  has  been  too  limited  to  provide  an 
assessment  of  the  full  potential  of  this  otherwise  promising  tool.  A 
distribution  copy  was  obtained,  and  the  tool  was  introduced  to  attendees  at  a 
mini-workshop.  Since  the  initial  reaction  of  users  has  been  favorable,  after 
further  experience  with  CLIPS  has  been  gained,  a  more  extensive  workshop 
featuring  this  tool  will  be  conducted. 

2.4  TESTING  AI  ISSUES 

This  effort  was  undertaken  as  a  survey  of  existing  and  proposed 
techniques  for  testing  kncwledge-based  systems  (KBS) .  The  intent  of  the 
survey  was  to  identify  available  techniques,  assess  their  relationship  to 
currently  defined  software  quality  factors,  and  make  recaimendations  for  their 
development  and  application. 

Specific  achievements  this  year  have  included;  initial  survey  of 
existing  and  proposed  techniques;  construction  of  a  data  base  reflecting 
technique  to  quality  factor  relationship;  update  of  a  previously  initiated 
bibliography  data  base  of  materials  related  to  such  techniques;  participation 
in  organization  and  conduct  of  a  workshop  on  validation  and  testing  of  KBS 
conducted  at  the  1989  International  Joint  Conference  on  Artificial 
Intelligence  (UCAI-89) ,  to  include  acceptance  of  two  papers  for  publication 
in  the  workshop  proceedings. 

The  papers  submitted  covered  an  initial  examination  of  the  applicability 
of  software  reliability  models  investigated  in  previous  methodology  efforts  to 
KBS  and  a  suninary  of  the  initial  testing  technique  survey  results. 

2.4.1  State  of  the  Art. 

In  the  past  three  to  four  years  there  has  been  a  significant  increase  in 
efforts  devoted  to  development  of  approaches  and  techniques  for  verification, 
validation,  and  testing  (W&T)  of  KBS.  Three  broad  categories  of  effort  have 
been  identified  to  date.  The  first  category  consists  of  those  projects  aimed 
at  defining  the  KBS  life  cycle  and  the  role  and  form  of  W&T  appropriate 
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within  that  context.  The  second  category  consists  of  projects  aimed  at 
developing  high-level  KBS  system  or  subsystem  assessments  from  seme 
combination  of  objective,  external,  performance  measures  and  subjective 
performance  ratings.  The  last  category  consists  of  projects  aimed  at 
development  of  detailed  and  generally  automated  procedures  for  measurement  of 
technical  characteristics  of  KBS. 

Projects  in  the  first  category  have  been  cxily  superficially  examined. 
Since  testing  done  by  USAEPG  occurs  at  specified  points  in  an 
externally-specified  life  cycle,  projects  of  this  type  have  limited 
applicability.  Their  primary  contribution  is  in  identification  of 
characteristics  and  criteria  for  KBS  evaluation,  and,  in  seme  cases,  of 
applicable  techniques. 

Projects  in  the  second  category  constitute  the  bulk  of  techniques 
immediately  available  for  application.  These  techniques  are  drawn,  with 
little  or  no  alteration,  from  W&T  procedures  for  decision  support  and  command 
and  control  systems.  They  require  very  little  tailoring  for  application  to 
KBS,  and  in  many  cases  parallel  techniques  in  use  new  at  USAEPG.  These 
techniques  suffer  from  the  drawback  of  being  oriented  towards  evaluation  of 
operational  effectiveness,  and  provide  little,  if  any,  of  the  technical 
specificity  required  for  developmental  testing  or  reliability,  availability, 
and  maintainability  (RAM)  assessment. 

Projects  in  the  third  category  have  the  greatest  potential  for 
application  to  developmental  and  RAM-related  testing.  Most  of  them  focus  on 
static  analysis  of  a  KBS  knowledge  base,  although  a  few  do  or  soon  will 
include  consideration  of  inference  engine  characteristics  as  well.  All  of 
these  projects  suffer  from  three  principle  drawbacks.  All  are  narrowly 
focused  on  a  specific  knowledge  representation,  in  a  manner  analogous  to  the 
limitation  of  many  static  analysis  tools  for  conventional  software  to  a  single 
language,  or  even  a  single  dialect.  All  but  two  of  the  tools  found  thus  far 
are  research  efforts  and  not  generally  available  production  quality  tools. 

All  but  three  of  the  tools  exist  independent  of  the  development  and 
maintenance  environment,  and  hence  require  additional,  ad  hoc  procedures  to 
obtain  the  necessary  source  or  other  output  for  their  application. 

The  allocation  of  each  of  the  techniques  examined  to  one  or  more  software 
quality  factors  leads  to  the  overall  KBS  W&T  state  of  the  art  rating  given 
belcw: 

Factor 
Correctness 
Reliability 
Efficiency 
Integrity 
Usability 
Maintainability 
Testability 
Flexibility 
Portability 
Reusability 
Interoperability 
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The  more  detailed  survey  results  are  included  in  the  copy  of  the  workshop 
paper  (reference  3) . 

2.4.2  Future  Work. 

The  current  effort  of  collecting  technique  documentation  and  entering  the 
techniques  in  the  project  data  base  cross-referenced  by  software  quality 
subfactors  will  continue.  The  fined  product  is  to  be  a  collection  of 
references  indexed  by  subfactor  which  could  aid  a  test  officer  in  identifying 
those  techniques  applicable  to  a  KBS  embedded  in  an  item  submitted  for  test. 
The  ultimate  purpose  of  the  collection  would  be  the  drafting  a  set  of  Test 
Operating  Procedures  (TOPs)  for  embedded  KBS,  although  actual  creation  of  the 
TOP  will  be  dependent  on  further  analysis  and  automation  of  selected 
procedures. 


As  indicated,  in  order  to  create  TOPs  for  KBS  testing  it  will  be 
necessary  to  further  analyze  and  define  a  selected  set  of  techniques  and  to 
develop  tools  for  their  application.  This  will  also  require  identification  of 
KBS  development  tools  enployed  in  those  projects  likely  to  develop  systems  or 
prototypes  to  be  submitted  to  USAEPG  for  test  in  order  to  allow  development  of 
inplementing  software  for  the  selected  techniques. 

Logical  candidates  for  initial  implementation  include  the  tests 
incorporated  in  Lockheed's  Expert  Validation  Assistant  (EVA) ,  the  matrix 
techniques  of  Aerospace  Corporation,  and  the  complexity  metric  proposed  by 
Structured  Systems  and  Software.  These  are  identified  and  references  are 
provided  in  the  referenced  workshop  paper. 

2.5  AVAILABILITY  OF  TOOLS 

Most  of  the  tools  developed  during  the  investigation  are  available  to 
other  Government  agencies  or  their  contractors.  For  current  information  on 
the  availability  of  a  tool,  contact: 

Bob  Harder 

Chief,  USAEPG  AI  Office 
AOTOVON  821-8183/8187 
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28  August  1987 

METHODOLOGY  INVESTIGATION  PROPOSAL 

1.  TITLE.  AI  Test  Officer  Support  Tool 

2.  INSTALLATION  OR  FIELD  OPERATING  ACTIVITY.  US  Anny  Electronic  Proving 
Ground,  Fort  Huachuca,  Arizona  85613-7110. 

3.  FRINCIPAL  INVESTIGATOR.  Mr.  Robert  Harder,  Software  and  Interoperability 
Division,  STEEP-ET-S,  ALTOVON  821-8187. 

4.  BACKGROUND.  Test  design  and  planning  for  modem  cPi  systems  require 
familiarity  with  a  number  of  test  operating  procedures  (TOPs)  as  well  as 
detailed  knowledge  of  specific  test  tool  capabilities.  A  wide  variety  of 
tests  must  be  designed,  planned,  and  scheduled  in  order  to  efficiently  conduct 
testing.  Interrelationships  among  test  groups  and  tools,  ccnmon  data 
requirements  and  data  reduction  and  analysis  requirements,  lead  time  to 
prepare  instrumentation,  and  required  availability  of  the  test  item  must  be 
well  understood  in  order  to  efficiently  conduct  tests  within  allocated  time 
constraints. 

USAEPG  has  explored  the  feasibility  of  an  automated  system  to  support  the 
test  officer.  Using  Independent  Laboratory  In-house  Research  (IUR)  funds,  a 
prototype  system  was  developed  using  AI  technology.  The  prototype  addressed 
tests  performed  by  the  Simulation  and  Interference  Branch  primarily,  but  could 
be  expanded  for  other  test  areas. 

.  3  .  . 

5.  PRORTFM-  Testing  C  I  systems  involves  designing  and  planning  tasks  which 
are  becoming  increasingly  coup  lex.  Advances  in  technology  such  as 
microprocessor  design,  distributed  real-time  architectures,  artificial 
intelligence,  and  electro-optics  are  appearing  in  new  Ci  developments.  While 
this  sophisticated  technology  offers  benefits  to  the  developer,  it  is  becoming 
a  considerable  burden  to  the  tester.  Test  officers  are  required  to  identify 
appropriate  test  methods  and  associated  instrumentation  and  data  acquisition 
requirements  for  each  emerging  technological  area.  This  requires  a  level  of 
expertise  which  is  rarely  found  in  any  one  individual.  Besides  being 
distributed  among  individuals,  and  therefore  less  available,  this  hard-earned 
expertise  is  frequently  lost  to  the  organization  because  of  personnel 
reassignment  or  attrition. 

6.  OBJECTIVE.  To  improve  test  methodology  by  providing  the  test  officer  with 
an  automated  support  tool. 

7.  MISSION  AREA(S)  SUPPORTED.  All  DA  mission  areas  for  systems  containing 
embedded  computer  resources  (ECR)  are  supported.  The  "Big  5"  program 
categories  (( T,  RSTA,  etc.)  are  accommodated  by  the  nonsystem-specific  nature 
of  the  methodology. 


AX  Test  Officer  Support  Tool  (Cont) 


a.  Summary.  The  investigation  will  draw  upon  previous  ILER  efforts  by 
expanding  the  level  of  detail  and  the  scope.  The  result  will  be  an  enhanced 
tool  supporting  the  test  officer  in  specific  domains  such  as  electromagnetic 
ocnpatibility,  software  testing,  and  general  test  mechanisms.  Other  domain 
categories  will  be  explored  as  time  permits. 

b.  Detailed  Approach.  The  USAEFG  will: 

(1)  Extract  and  codify  knowledge  from  cognizant  individuals  in  fields 
including  electromagnetic  and  software  testing. 

(2)  Examine  other  test  areas  to  identify  tests  performed,  responsible 
branches,  test  instrumentation  capabilities,  and  characteristic  test 
requirements.  Carmonality  among  these  various  factors  will  be  identified  to 
form  a  framework  which  will  icxxnmodate  all  test  functions,  instrumentation , 
and  resources.  Following  implementation  of  the  generalized  framework, 
specific  test  areas  (knowledge  domains)  will  be  analyzed  in  depth  and 
incorporated  into  the  tool. 

c.  Final  Product  (s) . 

(1)  An  AI  test  officer  support  tool  with  enhanced  capability — more 
"smarts"  in  the  existing  area  of  coverage,  and  additional  test  areas  covered. 

(2)  Requirements  and  reccranendations  for  automation  of  test  design 
and  planning  functions. 

d.  Coordination.  Extensive  coordination  with  the  varicus  test  groups  of 
the  USAEFG  is  an  inherent  characteristic  of  the  investigation.  To  the  extent 
that  test  areas  covered  overlap  the  areas  of  interest  of  other  I/PQAs, 
coordination  will  be  aoccrplished  through  existing  mechanisms  such  as  the 
TEOCM  Software  Technical  Ccrmittee  (TSOTEC) . 

e.  Environmental  Impact  Statement.  Execution  of  this  task  will  not  have 
an  adverse  impact  on  the  quality  of  the  environment. 

f.  Health  Hazard  Statement.  Execution  of  this  task  will  not  involve 
health  hazards  to  personnel. 

9.  JUSTIFICATION/IMPACT 

a.  Association  with  Mission.  This  investigation  directly  supports 
USAEFG' s  mission  relative  to  test  and  evaluation.  Providing  test  officers 
with  automated  support  tools  will  improve  the  efficiency  and  effectiveness  of 
testing. 

b.  Association  with  Methodoloav/Instrumentation  Program.  This  project 
supports  thrusts  of  the  TEOCM  Methodology  Program  to  improve  the  quality  of 
testing  as  well  as  test  process. 

pabilitv.  Limitations.  Improvement,  and  Impact  on  Testing  if 


c.  Present  Cai 
Approved. 
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(1)  Present  Capability.  TOPs  and  guidelines,  such  as  the  USAEPG  Test 
Officers  Handbook,  provide  static  information  on  test  methods  and  checklists 
for  test  planning  purposes. 

(2)  t .imitations.  Current  guidelines  often  do  not  provide  the  level 
of  detail  required  for  optimized  application  of  scarce  test  resources.  Also, 
the  information  is  static;  status  of  test  instrumentation,  competition  for 
resources  among  different  test  items,  and  the  impact  of  not  performing  some 
test  (or  lack  of  test  material  such  as  certain  documentation)  is  poorly 
handled  unless  the  test  officer's  experience  has  included  similar  situations. 

(3)  Improvement.  Using  AI  techniques  to  develop  a  support  tool  can 
provide  the  test  officer  sufficiently  detailed  and  flexible  guidelines. 

Beside  being  adaptable  to  the  needs  of  a  specific  test  item  and  current  with 
respect  to  test  instrumentation  availability,  the  proposed  approach  would  be 
sensitive  to  data  requirements  and  be  able  to  anticipate  the  impact  if  tests 
are  not  performed.  Supported  over  time,  such  a  tool  could  accumulate 
expertise  which  is  presently  distributed  and  too  frequently  lost. 

(4)  Impact  on  Testing  if  not  Approved.  The  expertise  required  of 
test  officers  is  rapidly  expanding  in  scope  as  innovative  technologies  are 
increasingly  enployed  by  developers.  The  corresponding  increase  in  complexity 
of  test  methods  and  instrumentation  demands  a  cxmnensurate  improvement  in 
support  tools  if  test  resources  are  to  be  effectively  and  efficiently  used. 
Also,  without  permanent  storage  and  readily  available  access  to  "lessons 
learned",  the  corporate  memory  of  an  activity  suffers  each  time  an  experienced 
individual  leaves  the  organization. 

10.  dotjar  SAVINGS.  NO  directly  supportable  dollar  savings  can  be  projected 
at  this  time.  Indirect  benefits  include  improving  the  quality  of  testing  and 
evaluation  leading  to  improved  quality  of  fielded  systems.  Equally  difficult 
to  quantify  is  the  retention,  concentration ,  and  increased  availability  of 
expertise,  which  is  potentially  a  significant  amount. 
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a.  Financial. 


Dollars  (Thousands)  Dollars  (Thousands) 
_ FY88 _  _ FY89 _ 


In-House 

(Xrt-of-House 

In-House  (Xrt-of-House 

Personnel  Compensation 

10.0 

12.0 

Travel 

3.5 

4.0 

Contractual  Support 

84.5 

42.5 

Materials  &  Supplies 

2.0 

1.5 

Subtotals 

15.5 

84.5 

17.5 

42.5 

FY  Totals 

100.0 

60.0 

b.  Explanation  of  Cost  Categories. 

(1)  Personpc  jmpensation.  This  cost  represents  compensation 
chargeable  to  the  investigation  for  using  technical  or  other  civilian 
personnel  assigned  to  the  investigation . 

(2)  Travel.  This  represents  costs  incurred  while  visiting 
government  and  industry  facilities. 

(3)  Contractual  Support.  Performance  of  the  investigation  will  be 
accomplished  with  resources  provided  under  an  existing  support  contract. 

c .  Obligation  Plan  fFY891 . 

-FQ  1  2  3  4  Total 

Obligation  Rate  45.0  5.0  5.0  5.0  60.0 

(Thousands) 

d.  Man-Hours  Required. 


In-House: 


Contract: 


AI  Test  Officer  Support  Tool  (Cont) 


12.  ASSOCIATION  WITH  TOP  PROGRAM.  This  investigation  will  not  directly 
produce  a  TOP.  However,  various  TOPS  may  require  review  and  revision  based  on 
the  findings. 

FDR  THE  CCM1ANEER: 


(signed) 

ROBERT  E.  REINER 
Chief,  Modernization  and 
Advanced  Concepts  Division 
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Domain-specific  references  are  available  upon  request. 
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APFENDIX  C.  ACRONYMS  AND  ABBREVIATIONS 


ADP . 

AI . 

AMC . 

AR . 

ASCII. . . 

ASL . 

BUD2. . . . 

CJI . 

CLIPS. . . 
CPEA. . . . 

DA . 

DoD . 

DTP . 

ECE . 

]=2> . 

EVA . 

EVA . 

EXSYS . . . 
I/PQA. . . 
UCAI-89 
ILIR. . . . 

KBS . 

MET . 

MS  DOS.. 
NASA.... 

PT . 

RAM . 

REC . 

RSTA. ... 

SAA . 

SBIR. . . . 

SEC . 

S&I . 

SOP . 

STARS. . . 
TEOCM... 

TOP . 

TPD . 

ISOTEC. . 
TIES. . . . 

UAV . 

USAEPG. . 
W&T. . . . 


Autanated  Data  Processing 
Artificial  Intelligence 
United  States  Army  Materiel  Command 
Army  Regulation 

American  Standard  Code  for  Information  Interchange 

Atmospheric  Sciences  Laboratory 

Budget  Spreadsheet  Analysis  Aid  Expert  System 

Contend,  Control,  and  Ccmunications 

Contend,  Control,  Ccnmmications  and  Intelligence 

C  language  Integrated  Production  System 

Contract  Performance  Evaluation  -  Advisor  Expert  System 

Department  of  the  Army 

Department  of  Defense 

Detailed  Test  Plan 

Embedded  Corputer  Resources 

Expert  System  Selector 

Environmental  Impact  Assessment  Expert  System 

Expert  Validation  Assistant  (Lockheed  AI  test  tool) 

Expert  System  Development  Package 

Installation  Field  Operating  Activity 

1989  International  Joint  Conference  on  AI 

Independent  laboratory  Inhouse  Research 

Knowledge  Based  Systems 

Meteorological  Expert  System 

Microsoft  Disk  Operating  System 

National  Aeronautics  and  Space  Administration 

Physical  Training 

Reliability,  Availability,  and  Maintainability 
Record  of  Environmental  Consideration 
Reconnaissance,  Surveillance,  and  Target  Acquisition 
Software  Analyst's  Assistant  Expert  System 
Small  Business  Innovative  Research 
Security  Expert  System 

Software  and  Interoperability  Division  (USAEPG) 

Standard  Operating  Procedure 

Software  Technology  for  Adaptable,  Reliable  Systems 

United  States  Army  Test  and  Evaluation  Command 

Test  Operating  Procedure 

Test  Plan  Drafter 

TEOCM  Software  Technical  Committee 

Tape  Test  Expert  System 

Unmanned  Aerial  Vehicle 

United  States  Army  Electronic  Proving  Ground 
Verification,  Validation,  and  Testing 


C-l 


This  page  intentionally  blank 


02 


APPENDIX  D.  FREVTOUS  APPLICATIONS 

This  section  contains  descriptions  of  the  major  prototype  expert  system 
applications  previously  reported.  They  are  included  here  to  provide  a  summary 
of  prior  results  without  having  to  reference  other  documents. 

1.  Test  Plan  Drafter. 

1.1  Purpose/Goals. 

The  near-term  goal  of  the  TPD  is  to  automate  the  current  manual  assembly 
of  boilerplate  for  an  initial  draft  of  a  detailed  test  plan  (DTP) .  This  is  a 
time-consuming  effort  consisting  of  much  cut-and-paste  work  from  old  test 
plans,  but  little  real  intellectual  effort.  It  is  intended  to  result  in  a 
stxawman  version  for  distribution  to  specific  subtest  domain  experts  for 
further  editing. 

A  longer  term  goal  of  the  effort  is  to  create  a  prototype  knowledge 
infrastructure,  i.e. ,  a  structure  for  centralized  maintenance  of  both  specific 
information  pertaining  to  a  given  test  and  general  information  needed  in  test 
planning. 

1.2  Dcmain/Expertise. 

The  initial  knowledge  acquisition  for  TPD  involved  determining  the 
structure  and  composition  of  a  DTP.  This  is  specified  in  part  by  applicable 
regulation.  Further  detail  was  provided  through  review  of  local  policy  and 
interviews  with  test  officers  and  with  USAEPG's  Technical  Publications 
Division  personnel  responsible  for  preparing  and  publishing  test  plans. 

Subsequent  efforts  involved  acquisition  of  previously  drafted 
boilerplates  for  specific  subtests  and  review  of  Army,  AMC,  TECDM,  and  USAEPG 
publications  to  refine  the  knowledge  of  test  plan  composition  and  of  the 
overall  test  and  evaluation  process  in  the  Army.  This  latter  knowledge,  in 
addition  to  aiding  in  understanding  the  test  planning  process,  is  essential  to 
realizing  the  desired  training  benefits  from  use  of  the  intended  tool. 

1.3  Requirements. 

The  general  requirements  laid  on  all  the  application  efforts  selected 
were  that  they  be  of  wide  utility  and  also  aid  in  training  of  personnel. 
Requirements  specific  to  TPD  were  that  it  reduce  the  manual  and  telephonic 
work  required  to  reach  the  strawman  stage  for  a  DTP,  that  it  provide 
information  on  test  plan  structure  and  component  descriptions,  and  that  it 
assist  the  novice  test  officer  in  understanding  the  test  and  evaluation 
process.  Requirements  added  during  the  prototype  development  were  that  it 
assist  in  draft  DTP  preparation  and  in  the  mechanics  of  the  DTP  review 
process. 

A  requirement  of  the  TPD  infrastructure  was  that  it  be  maintainable  in  a 
form  accessible  to  a  broad  range  of  offices.  For  this  reason,  the  tool 
selected  to  create  and  maintain  these  components  should  be  one  that  is  widely 
available  and  understood  by  personnel  not  necessarily  involved  in  or  familiar 
with  expert  system  or  AI  development. 
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1.4  Description.  The  current  TPO  prototype  performs  four  functions: 

a.  It  provides  a  simple  mechanism  for  entering  and  maintaining  standard 
information  pertinent  to  a  specific  test. 

b.  It  creates  a  partial  strawman  DTP  from  the  information  entered. 

c.  It  provides  background  information  on  the  test  and  evaluation  process 
in  general,  as  well  as  on  specific  ccrponents  of  a  test  plan. 

d.  It  provides  a  mechanism  for  preparing  a  draft  DTP  and  limited 
assistance  in  review  thereof. 

1-5  Design/ Development  characteristics. 

The  primary  development  environment  selected  for  TFT)  was  dBASE  III.  This 
environment  allows  attainment  of  the  infrastructure  goals  without  having  to 
retrofit  the  infrastructure  at  a  later  time.  The  Al-related  tools  used  are 
HYPE,  from  Knowledge  Garden,  Inc. ,  and  EXSYS  (Expert  System  Development 
Package) ,  obtained  from  California  Intelligence,  Inc.  HYPE  allows  use  of  the 
hypertext  paradigm  for  help  and  explanation  functions.  EXSYS  allows 
development  of  expert  system  components.  One  further  tool,  DOCUCCMP,  from 
Advanced  Software,  Inc.,  provides  a  document  comparison  facility  for 
identifying  changes  made  to  a  standard  subtest  to  tailor  it  for  a  specific 
system.  This  is  the  limited  assistance  provided  in  draft  DTP  preparation  and 
in  the  mechanics  of  the  DTP  review  process  (section  1.3) . 

The  initial  TPD  prototype  consists  entirely  of  the  dBASE  III  and  HYPE 
files  and  software  deeding  with  creation  and  maintenance  of  test  plan 
information,  and  associated  help  and  explanation  files.  The  dBASE  III 
application  software  drives  the  application,  invoking  HYPE  and  DOCUCCMP  where 
appropriate. 

1*6  Benefits/Use. 

TPD  is  designed  to  be  used  primarily  by  a  lead  test  officer  for  a 
specific  system,  to  assist  in  preparing  a  strawman  DTP.  Another  potential 
user  is  the  manager  evaluating  a  proposed  test  outline  for  a  potential 
customer. 

The  current  TPD  prototype  is  installed  in  the  Command  and  Control 
Division  of  the  Ccnmand,  Control,  and  Ccrmuni  cations  (C r)  Test  Directorate. 

It  has  been  used  in  production  of  severed  strawman  DTPs,  and  the  current  users 
have  made  several  suggestions  for  improvement. 

The  most  obvious  benefit  to  be  gained  from  the  TPD  is  time.  Current 
users  and  others  to  whan  TPD  has  been  demonstrated  indicate  that  the  current 
manual  method  of  strawman  draft  plan  composition  can  take  from  two  days  to  two 
weeks.  The  TPD  output  is  available  within  15  to  30  minutes.  While  the 
resulting  product  is  not  complete,  it  accounts  for  perhaps  as  much  as  30  to  50 
percent  of  the  content  of  such  a  strawman.  Seme  increase  in  this  percentage 
will  accrue  frcm  growth  in  the  archive  of  standard  subtests,  while  sane  must 
await  implementation  of  further  planned  functions. 
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Less  obvious  is  the  training  and  standardization  benefit.  The  hypertext 
provided  with  TPD  makes  available  to  the  novice  much  information  previously 
available  only  by  laborious  searching  through  assorted  regulations  and 
pamphlets.  It  also  indicates  sources  for  further  information.  Moreover, 
while  in  the  past  the  differing  backgrounds  and  levels  of  experience  among 
test  officers  have  sometimes  led  to  irregularities  in  DTP  and  subtest  format, 
more  widespread  use  of  a  single  tool  offers  the  premise  of  improved  adherence 
to  TEOCM  and  local  guidance  with  less  administrative  review  and  rewriting 
effort.  This  will  allow  test  officers,  test  engineers,  and  managers  to  devote 
more  of  their  time  and  effort  to  substantive  test  issues. 

1.7  Development  Status.  TPD  prototype  functionality  is  roughly  70  percent 
complete.  Addition  of  the  expert  system  components  for  subtest 
recommendations  and  coordination  requirements  will  bring  the  system  to  a  level 
that  will  permit  initial  formal  evaluation  of  its  utility.  This  will  require 
making  the  tool  more  widely  available  to  test  officers,  which  will  in  turn 
require  additional  copies  of  supporting  software. 


1.8  Future. 

Given  that  the  TPD  proves  worthy  of  continued  development,  three  major 
lines  of  development  are  foreseen.  The  first  is  expansion  and  refinement  of 
the  kncwl edge-based  portion  of  the  system,  i.e. ,  the  hypertext  and  expert 
system  components.  These  offer  considerable  potential  for  expansion  into 
expert  test  planning  areas,  such  as  costing  and  scheduling,  as  well  as 
tutorial  and  advisory  components  for  test  officer  training. 

The  second  area  involves  the  conventional,  infrastructure  component.  It 
is  important  here  to  note  only  that  this  effort  has  created  a  skeletal 
infrastructure  in  conventional  code  to  investigate  the  maintenance  and 
integration  problems  that  might  arise. 

The  third  line  of  development  involves  support  tool  integration.  The  use 
of  a  conventional  basis  for  this  tool,  and  development  of  a  standard  shell  for 
passing  the  information  contained  in  a  test-specific  data  base  to  an  expert 
system  component,  constitutes  an  example  of  one  integration  approach.  Further 
refinement  of  this  mechanism  and  comparison  with  others,  is  essential  to  allow 
integration  of  the  support  tools  into  a  single  package  for  use  by  the  test 
officer. 

2.  Environmental  Impact  Assessment  Expert  System. 

2.1  Purpose/Goals .  The  purpose  of  EVA  is  to  assist  the  test  officer  and 
environmental  personnel  in  collecting  accurate  environmental  information 
during  the  early  planning  phases  of  test  activities,  and  in  making  appropriate 
reoarmendations  based  on  characteristics  of  the  proposed  activities.  Specific 
goals  of  the  system  were  to: 

a.  Identify  tests  with  minimal  or  no  environmental  impacts,  and 
streamline  the  documentation  process. 

b.  Identify  possible  environmental  impacts  and  the  resources  that  could 
be  affected  (e.g. ,  water,  wildlife,  cultural,  historical) . 
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c.  Inprove  the  quality,  detail,  and  timeliness  of  information  provided 
to  environmental  personnel  during  the  initial  stages  of  a  test  project. 

d.  Incorporate  environmental  information  into  the  initial 
decision-making  stages  of  a  project. 

e.  Guide  activity  proponents  through  the  environmental  assessment 
process,  and  list  points  of  contact  for  action  items  and  regulatory 
requirements. 


2.2  Dcmain/Expertise. 

The  domain  of  EVA  covers  that  area  of  knowledge  required  to  identify 
potential  environmental  impacts,  recognize  categorical  exclusions  from  the 
rules  for  certain  damaging  activities,  and  perform  a  preliminary  screening  to 
determine  the  probable  environmental  documentation  requirements.  This 
expertise  resides  with  the  USAEPG  environmental  quality  coordinator  and 
environmental  specialists  attached  to  the  post  garrison.  These  experts  in 
turn  consult  specialists  in  more  narrow  domains  when  necessary. 

As  EVA  evolved  through  various  prototype  stages,  additional  information 
from  documented  sources  was  incorporated  into  the  design.  This  information 
consisted  more  of  quantitative  impact  factors,  rather  than  intuitive  knowledge 
about  the  domains.  The  inferences  about  this  data  were  supplied  by  the  human 
domain  experts. 

At  the  end  of  prototype  development  the  following  sources  had  been  used 
in  generating  the  data  bases  and  rules  of  EVA: 

a.  U.S.  Army  Corps  of  Engineers 

(1)  Construction  Engineering  Research  laboratory  reports 

(2)  Archaeologist 

b.  U.S.  Department  of  Agriculture,  Soil  Conservation  Service 

c.  U.S.  Army  Environmental  Hygiene  Agency 

d.  Fort  Huachuca 

(1)  Forester 

(2)  Wildlife  Biologist 

(3)  Fish  Biologist 

(4)  Environmental  Specialist 

Much  credit  is  due  the  post  environmental  specialist  for  identifying 
sources  of  information  and  eliciting  knowledge  from  subdomain  experts.  This 
effort  exceeded  the  scope  of  the  normal  participation  of  an  expert,  and  aided 
tremendously  in  knowledge  acquisition  activities. 

2.3  Requirements. 

USAEPG  is  required  to  conform  to  federal  and  state  environmental 
regulations  as  well  as  Army  and  DoD  policy  in  these  matters.  Every  proponent 
of  an  exercise  or  test  at  Fort  Huachuca  is  required  to  address  the 
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environmental  issues  associated  with  the  activity.  USAEPG  test  officers  have 
the  additional  responsibility  for  assessing  potential  environmental  impacts 
for  any  activity  resulting  from  a  test  directive,  regardless  of  the  nature  of 
the  testing. 

The  result  of  the  preliminary  impact  assessment  is  a  record  of 
environmental  consideration  (REC) .  The  REC  documents  the  consideration  of 
environmental  impacts;  possible  outcomes  are  that  the  activity  is  adequately 
covered  by  existing  documentation ,  qualifies  for  an  established  categorical 
exclusion  or  other  exemption,  or  requires  an  environmental  assessment. 
Environmental  assessments  subsequently  result  in  a  "finding  of  no  significant 
impact"  or  indicate  that  an  extensive  environmental  impact  statement  is 
required. 

Most  of  USAEPG' s  activities  are  conducted  at  locations  specifically 
designated  and  documented  for  that  type  of  activity,  or  are  conducted  entirely 
within  an  enclosed  facility,  as  such  computer  simulation  and  modeling.  Thus, 
the  major  requirement  of  a  preliminary  environmental  screening  is  to 
discriminate  as  early  as  possible  between  typical  situations  requiring  little 
further  documentation,  and  those  requiring  a  significant  environmental  impact 
study. 

2.4  Description.  The  EVA  elicits  information  about  a  proposed  activity  from 
the  test  officer,  and  reaches  a  preliminary  conclusion  on  the  actions 
required.  It  then  generates  a  report  containing  action  items,  and  summary  and 
detail  characteristics  of  the  activity,  with  corresponding  environmental 
impacts.  Activities  which  have  already  been  documented  or  qualify  for  a 
categorical  exclusion  are  quickly  identified  (i.e.,  a  minimum  of  user  input  is 
required) ,  and  the  necessary  REC  report  is  generated.  For  activities  where 
the  potential  environmental  impact  is  greater,  the  user  may  elect  to  examine 
the  environmental  resources  most  affected  and,  if  possible,  modify 
character ist ics  of  the  proposed  activity  to  minimize  the  impact  and  associated 
documentation .  In  any  event,  information  from  the  report  is  used  by  the 
environmental  quality  cxnrdinator  in  cxrpleting  the  environmental 
requirements. 

2.5  Design/ Development  Characteristics . 

The  EVA  system  consists  of  an  expert  system  which  provides  the  user 
interface,  cxntains  the  rules  used  to  make  decisions,  generates  reports,  and 
interfaces  with  other  tools  for  additional  capabilities.  These  other  tools 
supply  such  functions  as  acxess  to  data  bases  and  graphic  display  of  map 
information.  Other  components  include  supporting  information  such  as  help, 
system  parameter,  map,  point-of-contact,  and  report  specification  files.  The 
expert  system  shell,  EXSYS,  allows  a  means  to  interface  with  the  other  tools 
and  files  so  efficiently  that  the  user  is  generally  unaware  of  the  individual 
components.  To  further  isolate  the  user  frcm  having  to  contend  with  directory 
structures  and  operating  system  oxmnands,  a  set  of  cxmmand  files  was  created 
to  simplify  the  installation  and  operation  of  EVA.  The  user  merely  enters  one 
command  to  run  the  system  and  display  and  print  the  results. 

The  main  expert  component  of  EVA  oxntains  about  120  IF-THEN  rules  in  the 
knowledge  base.  When  processed  by  the  EXSYS  inference  engine,  the  rules  serve 
to  collect  the  necessary  information  to  reach  the  fined  conclusion  on  the 
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environmental  impacts  of  the  proposed  activity.  Forward  chaining,  a  technique 
which  determines  how  the  rules  are  processed,  also  allows  scxne  control  over 
the  sequence  in  which  events  take  place.  Thus,  the  user  can  be  presented  with 
queries  in  the  same  relative  order,  even  though  the  knowledge  base  and 
supporting  data  bases  may  have  changed  from  previous  versions. 

Although  all  of  the  rules  may  apply  to  a  given  scenario,  oily  those  which 
rely  upon  unknown  information  will  request  the  user  to  enter  needed  data. 
Besides  background  information  such  as  project  number  and  description,  which 
are  always  requested,  firing  (processing)  of  the  rules  may  trigger  queries  on 
up  to  150  numerical  or  textual  variables,  and  up  to  35  multiple-choice 
questions.  For  example,  if  the  activity  will  include  aircraft,  then 
information  is  requested  on  the  number  of  aircraft,  number  of  flight  hours, 
and  time  of  day  and  altitude  of  the  flights.  Because  only  essential 
information  is  requested,  an  EVA  session  can  last  anywhere  from  5  to  45 
minutes. 

Part  of  the  development  philosophy  was  to  minimize  the  amount  of 
knowledge  to  be  included  in  the  rules  about  a  specific  installation. 
Information  on  the  location  of  sensitive  resources,  period  of  sensitivity  if 
not  constant  throughout  the  year,  and  qualitative  damage  factors  associated 
with  particular  activities,  were  placed  in  ten  data  bases.  These  data  bases 
were  designed  to  be  readily  understood  and  modified  by  the  domain  experts 
without  first  having  to  obtain  knowledge  engineering  skills.  Likewise,  user 
help  screens,  point-of-contact  information,  etc. ,  which  contained 
installation-specific  material,  were  kept  in  separate  files.  This  approach 
may  provide  a  ready  means  of  porting  the  system  to  other  installations,  but 
was  chosen  primarily  to  reduce  development  and  maintenance  costs.  Information 
contained  in  the  various  data  bases  and  files  could  have  easily  been  encoded 
into  rules,  and  sane  expert  development  packages  provide  the  capability  to  do 
just  that  when  fed  tabular  data.  The  problem  with  a  pure  expert  system 
solution,  with  all  of  the  knc*-  ^dge  embedded  in  rules,  concerns  the  size  of 
the  resultant  rule  base.  It  v  estimated  that  to  incorporate  the  knowledge 
in  the  data  bases  alone  into  i  as  would  add  another  three  to  four  hundred 
rules.  Further  development  ana  maintenance  of  such  an  unwieldy  knowledge  base 
would  have  significantly  impeded  progress,  with  no  known  advantages. 

2.6  Benefits/Use. 

EVA  offers  benefits  to  the  test  officer,  environmental  quality 
coordinator,  and  program  manager.  Test  officers  are  given  the  opportunity  to 
ocnpare  environmental  effects  of  different  activities  at  various  locations  and 
times.  With  little  prior  knowledge  of  environmental  concerns,  the  test 
officer  using  EVA  can  quickly  gain  an  appreciation  of  the  relative  impact  of 
various  activities  through  the  questions  asked,  the  associated  help  text,  and 
the  outcome  of  the  proposed  scenarios.  Less  experienced  test  officers  also 
benefit  from  the  action  items  and  notes  related  to  the  proposed  activity; 
e.g. ,  contacting  the  fire  marshal  and  filing  a  fire  plan  if  incendiary  devices 
are  used,  or  coordinating  tree  and  brush  removal  with  the  post  forester. 

These  serve  as  reminders  even  for  seasoned  test  officers,  and  both 
inexperienced  and  experienced  users  of  the  system  benefit  from  reduced 
paperwork  and  coordination. 
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EVA  does  not  make  complicated  environmental  decisions,  write 
environmental  assessments  or  environmental  inpact  statements,  or  replace 
environmental  personnel.  In  fact,  environmental  quality  coordinators 
themselves  can  use  EVA  to  refine  the  work  initiated  by  test  officers,  or  as  a 
method  of  automating  and  documenting  activities  in  a  standard  fashion.  Tests 
with  minimal  environmental  inpact  are  identified  with  a  savings  of  paperwork 
and  time.  Even  for  large  activities  not  fully  handled  by  EVA,  the  quality, 
consistency,  and  detail  of  information  presented  to  environmental  personnel  is 
greatly  improved.  Without  EVA,  many  preliminary  meetings  are  required  between 
the  test  officer  and  environmental  quality  coordinator,  merely  to  establish 
what  information  is  needed,  and  then  the  data  is  rarely  available  in  an 
organized  format. 

Sponsors  of  testing  activities  may  gain  the  most  from  the  use  of  EVA, 
albeit  somewhat  indirectly.  Because  extensive  environmental  documentation 
requirements  can  cause  lengthy  and  expensive  delays,  it  is  important  to 
identify  potential  impacts  as  early  as  possible,  and  develop  alternative  test 
scenarios  which  are  more  environmentally  benign.  Advance  warning  of 
potentially  expensive  activities,  such  as  disposal  of  hazardous  materials 
(e.g. ,  expended  batteries) ,  may,  if  given  in  time,  allow  implementation  of 
more  cost-effective  solutions. 

2.7  Development  Status.  EVA  is  currently  installed  on  several  microcomputer 
systems  at  Fort  Huachuca;  about  20  test  officers  have  been  formally  trained  in 
its  use.  Presently  the  system  is  in  an  evaluation  phase,  where  feedback  is 
being  obtained  concerning  its  use  in  test  operations. 

2.8  Future.  A  number  of  ideas  for  further  development  of  EVA  have  been 
preposed.  During  its  construction,  the  development  team  identified  a  number 
of  desirable  features  which  could  not  be  implemented  because  of  time 
constraints.  Other  valuable  ideas  emerged  from  the  test  officer  training 
sessions.  However,  the  actual  usefulness  and  benefits  to  be  realized  must  be 
determined  from  the  results  of  the  ongoing  evaluation.  Sane  of  the  more 
significant  limitations  and  improvements  to  be  considered  in  future  efforts 
are  the  following: 

a.  Some  of  the  knowledge  in  EVA  is  in  a  preliminary  state,  having  been 
added  to  determine  the  feasibility  and  desirability  of  certain  features  (e.g. , 
a  component  to  address  hazardous  materials) .  Those  features  deemed  desirable 
should  be  expanded,  along  with  the  rest  of  the  system,  into  a  fully 
operational  form. 

b.  The  potential  for  porting  the  system  to  other  installations  should  be 
explored  further.  This  would  require  an  initial  analysis  of  the  requirements 
of  other  installations,  to  see  if  enough  ccrmonality  exists  in  the  knowledge 
domains  to  make  this  approach  feasible.  Such  an  investigation  might  also  shed 
seme  light  on  the  ccrmonality  of  other  requirements,  such  as  test  resource 
management  and  safety. 

c.  The  prototype  system  has  the  limitation  that  cxily  one  map  area  can  be 
entered  as  the  location  of  activity.  Although  areas  may  be  arbitrarily 
defined  as  large  or  small  as  desired,  a  cumbersome  situation  occurs  with 
activities  consisting  of  100  or  more  sites  with  minimal  inpa  ct  at  each 
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location.  Even  smaller  activities  may  be  handled  better  if  multiple 
locations,  or  if  unrestricted  boundaries  are  allowed. 


d.  A  feature  which  would  allow  saving  all  of  the  input  information,  to 
be  used  later  to  examine  the  impact  of  different  test  scenarios,  is  desirable. 
Such  a  capability  was  partially  implemented,  but  had  to  be  disabled  because  of 
a  software  discrepancy  in  the  expert  system  tool.  Along  these  same  lines, 
many  users  expressed  the  desire  to  be  able  to  modify  an  entry  that  had  just 
been  made.  Both  seem  to  be  necessary  features  for  practical  use  in  an 
operational  environment. 

e.  Most  of  the  data  bases  of  EVA  are  indexed  by  location.  Geographic 
information  also  plays  an  important  part  in  many  other  functions  at  Fort 
Huachuca.  A  solution  to  many  of  these  needs  for  information  associated  with 
geographic  position  would  be  a  geographic  information  system.  This  is  also  a 
requirement  of  many  other  proposed  test  tools.  While  implementation  and 
maintenance  of  such  a  system  is  well  beyond  the  scope  of  this  investigation, 
the  potential  usefulness  is  great  enough  to  warrant  development  by  other 
means. 

f.  The  ac±ual  users  of  EVA  range  from  inexperienced  test  officers  to 
qualified  environmental  personnel.  Because  of  the  disparity  in  experience,  a 
system  tailored  to  a  given  skill  level  will  be  somewhat  frustrating  for  users 
of  a  different  level.  Experienced  users  quickly  tire  of  a  system  oriented 
tcward  the  novice,  while  inexperienced  users  may  find  a  system  written  for  the 
expert  to  be  much  too  difficult.  A  possible  solution  to  this  dilemma  was 
discovered  during  the  EVA  development,  but  too  late  to  fully  evaluate. 
Basically,  this  approach,  if  implemented,  would  call  for  multiple  levels  of 
rules,  help,  and  queries.  A  "don't  understand"  option  is  provided  on  higher 
level  queries.  When  invoked  by  the  novice,  this  option  fires  lower  level 
rules  which  elicit  a  number  of  simpler  details  from  the  user.  These  details 
are  then  formulated  by  the  lower  level  rules  into  facts  which  satisfy  the 
original ,  "difficult"  query.  Such  an  approach  is  best  implemented  on  mature 
knowledge  bases  because  of  the  growth  in  size  and  ocrmensurate  decline  in 
maintainability.  For  a  system  with  a  diverse  user  base,  further  examination 
of  this  technique  may  prove  useful. 

3.  Meteorological  Expert  System. 

3.1  Purpose/Goals .  The  Meteorological  Expert  System  (MET)  began  originally 
as  a  manual  paper  checklist  for  test  officers  to  use  in  preparing  for  upcoming 
tests  at  Fort  Huachuca.  It  is  designed  to  emphasize  the  need  for 
meteorological  data  in  planning  and  reporting  tests  within  USAEPG.  MET  also 
indicates  that  various  meteorological  measurements  and  advisories  are 
available  frcm  the  Atmospheric  Sciences  Laboratory  (ASL)  weather  station  at 
Fort  Huachuca,  and  from  other  sites  located  on  the  Fort  Huachuca  ranges. 

3.2  Domain/Expertise.  This  expert  system  deals  with  the  knowledge 
enocnpassing  meteorological  measurements  and/or  those  weather  events  which 
affect  test  operations  on  the  ground  or  in  the  atmosphere  where  testing  will 
take  place.  Generally  these  measurements  or  observations  are  provided  by  ASL. 

3.3  Requirements.  From  the  standpoint  of  the  test  officer,  the  need  for  an 
expert  system  on  weather  is  that  it  can  educate  and  inform  the  test  officer 
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about  meteorological  data  requirements  and  available  resources  for  a  test.  The 
need  for  such  data  acmes  primarily  when  the  test  will  be  conducted  outdoors. 
The  expert  system  will  make  clear  that  the  officer  will  need  to  have  weather 
predictions  before  the  test  in  order  to  plan  for  conditions  such  as  cold  or 
heat,  rain  or  snow,  and  wind  or  licflitning.  Weather  advisories  and  weather 
alerts  from  ASL  can  warn  the  test  officer  in  the  field  of  impending  sudden 
weather  changes  that  could  endanger  personnel  and  equipment. 

3.4  Description. 

The  MET  system  educates  the  test  officer  as  to  possible  weather-related 
needs,  and  informs  the  officer  chi  how  to  obtain  needed  measurements  to  prepare 
for  the  test,  how  to  run  the  test  more  effectively,  and  how  to  obtain  weather 
station  support  in  reporting  the  test  outcome. 

Measurements  and  predictions  of  temperature,  dew  point,  rain,  snow, 
thunderstorm  activity,  and  winds  in  the  lower  atmosphere,  may  be  needed. 
Predictions  may  be  needed  as  to  meteorological  conditions  such  as  sunspot 
activity  and  atmospheric  index  of  refraction.  MET  informs  the  test  officer 
whether,  during  on-site  test  activities,  weather  advisories  and  reports  of 
selected  meteorological  values  are  available  and  may  be  needed.  Also  the  ASL 
weather  station's  ability  to  support  test  reporting  is  covered. 

The  result  of  using  the  MET  system  is  that  the  test  officer  can  produce 
better  test  data  by  being  prepared  with  needed  meteorological  data,  both  in 
measurements  that  directly  supply  parameters  needed  in  the  calibration  of 
equipment  such  as  radar,  and  in  supplying  measurements  for  the  test,  as  well 
as  weather  advisories  that  assist  in  day-to-day  running  of  test  operations. 

Without  MET,  the  test  officer  must  know  to  inform  ASL  of  test 
requirements  far  enough  in  advance  to  prepare  them  to  supply  information 
needed  for  the  test.  ASL  may  need  to  prepare  ahead  of  time  to  be  able  to  make 
measurements  during  the  test,  and  will  need  to  know  what  data  are  needed  for 
the  test  report.  ASL  can  supply  report s  of  the  meteorological  conditions  that 
existed  during  testing. 

3.5  Design/ Development  Characteristics. 

The  MET  system  is  ucn posed  of  a  series  of  questions  which  are  presented 
to  a  test  officer  from  within  the  EXSYS  shell.  The  questions  asked  in  this 
prototype  version  of  MET  determine,  for  example,  whether  lasers  will  be  used 
in  the  atmosphere,  whether  any  radar  or  unmanned  aerial  vehicles  (UAVs)  will 
be  used,  whether  personnel  and/or  equipment  will  be  in  the  field,  and  whether 
heavy  rain  or  snow  will  be  a  problem.  Fran  such  factors,  MET  can  then  advise 
that  meteorological  measurements  will  be  needed  to  support  these  activities. 
For  example: 

a.  Aerosol  density  in  the  atmosphere  or  optical  scintillation 
measurements  may  be  needed  for  a  test  involving  lasers. 

b.  Meteorological  data  used  in  radar  calibration  may  be  needed  for  a 
test  using  or  testing  radar. 

c.  Measurements  of  upper  air  winds  and  turbulence  could  be  needed  for 
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a  test  using  UAVs. 

d.  Weather  advisories  would  be  wise  to  have  during  test  activities. 

MET  automates  the  original  wsather/meteorological  checklist  into  a  system 
in  which  the  questions  are  presented  on  the  ccrputer  monitor  for  decision, 
help  is  provided  by  way  of  a  ccrputer-stored  text  file  for  each  question,  and 
the  answers  are  stored  in  ccrputer  memory  until  the  sessions  end,  when  a 
report  including  all  input  answers  is  produced.  The  report  is  displayed  on 
the  computer  monitor  and  printed  on  the  line  printer,  under  operator  control. 

3.6  Benefits/Use.  The  benefit  of  using  the  MET  system  is  that  the  test 
officer  becomes  better  informed  about  available  support  from  the  ASL  weather 
statical,  and  learns  what  weather  conditions  require  special  preparation.  The 
test  officer  can  then  more  likely  plan  the  test  so  as  to  produce  a  more 
accurate  result,  and  will  be  able  to  write  a  more  correct  and  informative 
report.  This  fill  adds  up  to  savings  in  time  and  money. 

3.7  Development  Status.  MET  has  been  developed  only  to  the  initial 
evaluation  stage.  In  this  prototype  version,  MET  has  been  placed  on  10 
microcomputers  in  the  USAEPG  and  ASL  offices  at  Fort  Huachuca,  so  as  to  be 
available  for  use  by  all  test  officers.  Statistics  on  system  usage  and 
cements  on  deficiencies  or  possible  improvements  have  not  yet  been  collected. 

3.8  Future.  After  evaluation,  the  MET  prototype  will  be  modified  to 
eliminate  any  discrepancies  found,  and  to  enhance  the  system's  capabilities  to 
better  serve  test  officer  needs.  Questions  will  be  improved  to  clarify  their 
meaning.  The  MET  help  file  will  be  changed,  as  needed,  to  make  explanations 
more  useful  to  the  user.  The  sequence  of  questions  presented  to  each  test 
officer  will  be  determined  by  previous  answers  to  prevent  redundancies. 
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U.S.  Anny  Materiel  Systems  Analysis  Activity 
ATIN:  AMXSY-CA 
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U.S.  Army  Test  and  Evaluation  cccmand 

ATIN:  AMSTE-TA-W 

ATIN:  AMSTE-TC-M 

ATIN:  AMSTE-TA 

ATIN:  AMSTE-TO 

Aberdeen  Proving  Ground,  MD  21005-5055 
Oormander 

Defense  Technical  Information  Center 
ATTN:  FDAC 
Cameron  Station 
Alexandria,  VA  22304-6145 

Germander 

U.S.  Army  Cold  Regions  Test  Center 

ATIN:  STECR-TM 

AFO  Seattle,  WA  98733-5000 

Conmander 

U.S.  Army  Combat  Systems  Test  Activity 
ATIN:  STECS-DA-M 

Aberdeen  Proving  Ground,  MD  21005-5000 
Conmander 

U.S.  Army  IXjgway  Proving  Ground 
ATIN:  STEDP-PO-P 
IXigway,  UT  84022-5000 

Conmander 

U.S.  Army  Electronic  Proving  Ground 

ATIN:  STEEP-TD 

ATIN:  STEEP-ET 

ATIN:  STEZP-DT 

ATTN:  STEEP-MD 
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Conmander 

U.S.  Army  Jefferson  Proving  Ground 
ATIN:  STEJP-TO-E 
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Pomander 

U.S.  Army  Tropic  Test  Center 

ATTN:  STETC-TD-AB  1 

APO  Miami,  FL  34004-5000 

Conmander 

U.S.  Army  White  Sands  Missile  Range 

ATIN:  STEWS -TE-A  1 

ATTN:  STEWS -TE-M  1 

ATIN:  STEWS -TE-0  1 

ATIN:  STEWS -TE-PY  4 

White  Sands  Missile  Range,  NM  88002-5000 

COnmander 

U.S.  Army  Yuma  Proving  Ground 

ATTN:  STEYP-MSA  2 

Yuma,  AZ  85634-5000 
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